Throttling

Throttling interface
Throttling

Introduction

If you need to throttle, Scrapfly gives you the ability to limit the pressure on an upstream website. You can throttle the concurrency (number of simultaneous requests) and the rate (maximum number of requests per window time).

See In Your Dashboard

Some websites monitor the traffic and apply a rate limit. Most of the time, you will need to throttle when you scrape content as an identified user (OAuth, JWT, token, and the like). In that case, changing IP will not affect the rate limit since you’re identified via your identity.

The throttler feature's idea is to respond to the need to manage throttling at a distributed level. Again, throttling is a well-known problem but costs time and effort to bring a production-grade system. It's integrated into Scrapfly API, ready and easy to use with no additional cost.

Scope

Throttles are scoped by environment and project.

By default Throttling is activated, you can disable it be specifying throttle=false. Throttles are matched according to the host pattern and priority. If there are many throttle matches, the higher priority is selected.

Rate Limiter

The rate limiter is using the sliding window algorithm. As soon as the request is out of windows, it's released. It means you smoothly retrieve your quota over time instead of waiting the whole period to recover the total quota.