Throttling

Throttling interface
Throttling

Introduction

If you need to throttle Scrapfly give you the possibility to limit the pressure on upstream website. You can throttle the concurrency (Number of simultaneous request) and the rate (Maximum number of request for on a window time).

See in your dashboard

Some website monitor the traffic and apply a rate limit. Most of the time you will need throttle when you scrape content as identified user (oauth, jwt, token, etc). It that case changing IP will not the affect the rate limit since your identified via your identity.

The idea behind the throttler feature, is to respond on the need to manage throttling at a distributed level. Again, throttling is a well known problem but cost time and effort to bring a production grade system. It's integrated to Scrapfly API, ready and easy to use with no additional cost.

Scope

Throttle are scoped by environment and project.

By default Throttling is activated, you can disable it be specifying throttle=false. Throttle are matched according the host pattern and priority. If many throttle match, the higher priority is selected.

Rate Limiter

Rate limiter is using sliding window algorithm, as soon as the request is out of windows, it's released. It's means, you smoothly retrieve your quota over time instead of waiting a the whole period to retrieve the full quota.