[Blog](https://scrapfly.io/blog)   /  [blocking](https://scrapfly.io/blog/tag/blocking)   /  [How to Rate Limit Async Requests in Python](https://scrapfly.io/blog/posts/how-to-rate-limit-asynchronous-python-requests)   # How to Rate Limit Async Requests in Python

 by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) Apr 10, 2026 3 min read [\#blocking](https://scrapfly.io/blog/tag/blocking) [\#python](https://scrapfly.io/blog/tag/python) 

 [  ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-rate-limit-asynchronous-python-requests "Share on LinkedIn") [  ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-rate-limit-asynchronous-python-requests&text=How%20to%20Rate%20Limit%20Async%20Requests%20in%20Python "Share on X") [  ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-rate-limit-asynchronous-python-requests "Share on Facebook")    

 

 

Summarize this article with

 [  ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-rate-limit-asynchronous-python-requests) [  ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-rate-limit-asynchronous-python-requests) [  ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-rate-limit-asynchronous-python-requests) [  ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-rate-limit-asynchronous-python-requests) [  ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-rate-limit-asynchronous-python-requests) 



   

When web scraping, we're often limited by the website's technical capabilities. We can't scrape too fast without being blocked or overwhelming smaller websites.

Asynchronous HTTP clients like [httpx](https://pypi.org/project/httpx/) allow us to easily make hundreds of requests per second but don't provide ways to fine-tune our scraping speed. So, in this quick tutorial, we'll be taking a look at how to rate-limit asynchronous HTTP connections to slow down our scrapers.

## Key Takeaways

Master async Python rate limiting with advanced throttling techniques, connection management, and performance optimization for comprehensive web scraping operations.

- Implement aiometer library for precise request rate limiting with concurrent connection control
- Configure httpx.Limits for connection pooling and concurrent request management
- Use asyncio.Semaphore for advanced throttling with custom rate limiting algorithms
- Implement request spacing and delay mechanisms to avoid overwhelming target servers
- Configure specialized tools like ScrapFly for automated rate limiting with anti-blocking features
- Monitor request patterns and implement adaptive throttling for optimal scraping performance

**Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime.





## Python httpx

HTTPX is the most popular asynchronous HTTP client in Python which can be installed using `pip install` terminal command:

shell```shell
$ pip install httpx
```



HTTPX supports both synchronous and asynchronous HTTP clients. It's unlikely that we need to throttle synchronous connections as they are very slow to begin so let's take a look at the throttling options we have for the asynchronous client.

To limit httpx client we can use the `httpx.Limit` object:

python```python
import httpx
session = httpx.AsyncClient(
    limits=httpx.Limits(
        max_connections=5  # we can change max connection count here
    )
)
```



However limiting scraping by connection count is very inaccurate - on slower websites 5 connections might only manage a few requests per minute while on fast ones it could reach hundreds of requests per minute.

To limit httpx-powered scrapers we need an additional layer that tracks requests themselves rather than connections. Let's take a look at the most popular throttling library - `aiometer`.

## Python aiometer

The most popular way to limit all asynchronous tasks in Python is [aiometer](https://pypi.org/project/aiometer/) which can be installed using the `pip install` terminal command:

shell```shell
$ pip install aiometer
```



Then we can schedule all of our requests to run through aiometer limiter:

python```python
import asyncio
from time import time

import aiometer
import httpx

session = httpx.AsyncClient()


async def scrape(url):
    response = await session.get(url)
    return response


async def run():
    _start = time()
    urls = ["https://httpbin.dev/html" for i in range(10)]
    results = await aiometer.run_on_each(
        scrape, 
        urls,
        max_per_second=1,  # here we can set max rate per second
    )
    print(f"finished {len(urls)} requests in {time() - _start:.2f} seconds")
    return results


if __name__ == "__main__":
    asyncio.run(run())

# will print:
# finished 10 requests in 9.54 seconds
```



In our small example scraper we used `aiometer.run_on_each` function to limit 10 scrape requests at 1 request per second. With this one command, we can throttle our scraper to exact requests per second speed!

## Scraping Faster Without Rate Limiting?

Some websites impose extremely low-speed limits for web scrapers making data collection impossible. To handle this [web scraping APIs](https://scrapfly.io/web-scraping-api) like ScrapFly can be used to scrape faster and avoid blocking.

ScrapFly offers dozens of different features that can help with scraper scaling, like:

- [Anti Scraping Protection Bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection)
- [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering)
- [190M Pool of Residential or Mobile Proxies](https://scrapfly.io/docs/scrape-api/proxy)

To use ScrapFly in Python install [ScrapFly-sdk](https://scrapfly.io/docs/sdk/python) package using the `pip install scrapfly-sdk` terminal command. Then targets can be scraped without the blocking:

python```python
from scrapfly import ScrapflyClient, ScrapeConfig
client = ScrapflyClient(
    key="YOUR SCRAPFLY KEY",
    max_concurrency=10,  # we can limit concurrent requests if needed
)
result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev/ip",
    # optional features like:
    # - we can select specific proxy country
    country="GB",
    # - and enable anti scraping protection bypass:
    asp=True,
    # see https://scrapfly.io/docs/scrape-api/getting-started for more
))
```





 

   [  Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents















 

  Table of Contents- [Key Takeaways](#key-takeaways)
- [Python httpx](#python-httpx)
- [Python aiometer](#python-aiometer)
- [Scraping Faster Without Rate Limiting?](#scraping-faster-without-rate-limiting)
 
    Join the Newsletter  Get monthly web scraping insights 

 

  



Scale Your Web Scraping

Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits.

  No credit card required  1,000 free API credits  Anti-bot bypass included 

 [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) 

 Not ready? Get our newsletter instead. 

 

 ## Related Articles

 [  

 python scaling 

### Web Scraping Speed: Processes, Threads and Async

Scaling web scrapers can be difficult - in this article we'll go over the core principles like subprocesses, threads and...

 

 ](https://scrapfly.io/blog/posts/web-scraping-speed) [  

 http python 

### Web Scraping with Python

Introduction tutorial to web scraping with Python. How to collect and parse public data. Challenges, best practices and ...

 

 ](https://scrapfly.io/blog/posts/web-scraping-with-python) [  

 http 

### What is HTTP 413 Error? (Payload Too Large)

HTTP status code 413 generally means that POST or PUT data is too large. Let's take a look at how to handle this.

 

 ](https://scrapfly.io/blog/posts/http-error-413-payload-too-large) 

  ## Related Questions

- [ Q What is Asynchronous Web Scraping? ](https://scrapfly.io/blog/answers/what-is-asynchronous-web-scraping)
 
  



   



 Bypass anti-bot protection automatically, **1,000 free credits** [Start Free](https://scrapfly.io/register)