Monitoring

Scrapfly offers a detailed real-time monitoring dashboard that logs all scrape requests and their results. This dashboard tracks all scrape request for the selected Scrapfly project and environment. It can be filtered and inspected for overall scraping performance:

See Your Monitoring Dashboard

Logs Retention
FREE DISCOVERY PRO STARTUP ENTERPRISE
Log retention 1 weeks Log retention 1 weeks Log retention 2 weeks Log retention 3 weeks Log retention 4 weeks
Screenshots, Debug, Cache belongs to log and inherit of the same retention - As soon as the log is deleted, they are also deleted

Filters

Filters allow you to sample or search logs from the monitoring section to investigate or check that everything works as expected.

Time Frame

Eight pre-configured time frames are available (past month, past week, past day, past three hours, past hour, past 30 minutes, past 15 minutes, past 5 minutes, past 5 minutes). You can also define an arbitrary time frame.

Dimensions

You can filter the following values:

Name Value Support Multiple Description
url string Yes Filter for URL, support glob operator with.
e.g: https://*.wikipedia.org/wiki/*
success bool true or false No Filter for success or failed request - includes >= 500 and network errors
domain string Yes Filter for a domain, the domain (TLD+n) includes a subdomain, support glob operator with
e.g.: *.google.com
root_domain string yes Filter for root domain (TLD+1), do not include subdomain, support glob operator with
e.g.; goo*.com
method string Yes Filter for method, supported values : GET, PUT, POST, PATCH
status_code int Yes Filter for status code
e.g: 401 or 502,504
cost int 1 Filter based of on cost spent on API Credits
origin string Yes Filter for origin, supported values : API, SCHEDULER
retries int No Filter for retries amount
duration int No Filter for duration
error_code string Yes Filter by error codes e.g: ERR::SCRAPE::SCENARIO_EXECUTION
errored bool true or false No Filter scrape having an error

For metrics that support multiple notations, simply separate with , like status=200,204

Chaining Filters & Multi Values

You can chain multiple filters with space.

You can filter over multiple values, in this case, OR operator is applied.

Operators

Following operators are supported :

  • = Equal
  • ! Not equal
  • > Greater than
  • >= Greater or equal to
  • < Lower than
  • <= Lower or equal to

Summary