Understand how timeout is working

You must configure your http client read timout to 155s to avoid any issue. Scrapfly will timeout according the strategy used.

Time management is crucial in web scraping in order to recover as fast as possible. Everything is budget and monitored to timeout the scrape.

Some scrapes are fast <5s, some other can require more time when rendering javascript ~25s or even more when using complex user scenario ~90s

Default Values

With retry=true (Scrapfly will auto retry on recoverable errors)

  • Scrape will timeout after 30s
  • The maximum timeout is 30s
  • 3 rounds of retries will be performed in case of issue - (90s)
  • With ASP enabled:
    • ASP can extend round of retry in order handle special case
    • ASP can consume the whole timeout budget (145s)

With retry=false (Scrapfly will not auto retry on recoverable errors)

Custom Timeout Value

You can specify your own timeout value with url parameter timeout=1500 (timeout parameter is in milliseconds)

  • If retry=true the maximum allowed is 30s
  • If retry=false the maximum allowed is 145s
  • If asp=true the minimum allowed is 30s
  • If rendering_js=true the minimum allowed is 30s
  • If javascript scenario is defined it can't be less than cumulated time required to run it - each step a budget time required

Related Errors

FAQ

  • Question: I want to run a user scenario that take at least 90s
  • Answer: Specify retry=false&timeout=120