Understand how timeout is working

You must configure your HTTP client read timout to 155s to avoid any issues. Scrapfly will manage the timeout according the strategy used. If you specify a custom timeout value, add 5s to your read timeout - screenshot, debug, cache add some overhead

Timeout configuration allow you set a deadline when you start a scrape. In that way you can ensure the scrape will not take more time than the defined timeout, Scrapfly will stop and return an error.

Time management is crucial in web scraping in order to recover as fast as possible. Everything steps are budgeted and tracked to prevent and recover from issue as fast as possible and provide the best reliability. Some scrapes are fast <5s, some other can require more time when rendering javascript ~25s or even more when using complex user scenario ~90s

To be able to customize a timeout, retry must be disable retry=false.

When Should I configure Timeout

If you are in the one of the following case:

timout diagram
Always +5s to your client read timeout when you customize the scrape timeout.
If you disable retry while using ASP, the default timeout is 30s. However, regarding some targeting that require are quite slow to pass, we recommend to increase the timeout to 60s as minimum. Below 60s, there is a high chance that on slow website or challenge, our system is not able to recover, rotate and bypass again. It will result by a blocked scrape on your end.

Usage Example

API Example

curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "retry=false" \
--data-urlencode "tags=player,project:default" \
--data-urlencode "timeout=15000"


key      = "" 
url      = "https://httpbin.org/anything" 
retry    = "false" 
tags     = "player,project:default" 
timeout  = "15000" 

Related Errors