If you need to throttle, Scrapfly gives you the ability to limit the pressure on an upstream website. You can throttle the concurrency (number of simultaneous requests) and the rate (maximum number of requests per window time).
Some websites monitor the traffic and apply a rate limit. Most of the time, you will need to throttle when you scrape content as an identified user (OAuth, JWT, token, and the like). In that case, changing IP will not affect the rate limit since you’re identified via your identity.
The throttler feature's idea is to respond to the need to manage throttling at a distributed level. Again, throttling is a well-known problem but costs time and effort to bring a production-grade system. It's integrated into Scrapfly API, ready and easy to use with no additional cost.
Throttles are scoped by environment and project.
Throttles are matched according to the host pattern and priority. If there are many throttle matches, the higher priority is selected.
The rate limiter is using the sliding window algorithm. As soon as the request is out of windows, it's released. It means you smoothly retrieve your quota over time instead of waiting the whole period to recover the total quota.
All related errors are listed below. You can see full description and example of error response on Errors section