[Blog](https://scrapfly.io/blog)   /  [httpx](https://scrapfly.io/blog/tag/httpx)   /  [How to Web Scrape with HTTPX and Python](https://scrapfly.io/blog/posts/web-scraping-with-python-httpx)   # How to Web Scrape with HTTPX and Python

 by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) Sep 17, 2026 11 min read [\#httpx](https://scrapfly.io/blog/tag/httpx) [\#python](https://scrapfly.io/blog/tag/python) 

 [  ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-python-httpx "Share on LinkedIn") [  ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-python-httpx&text=How%20to%20Web%20Scrape%20with%20HTTPX%20and%20Python "Share on X") [  ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-python-httpx "Share on Facebook")    

 

 

Summarize this article with

 [  ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-python-httpx) [  ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-python-httpx) [  ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-python-httpx) [  ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-python-httpx) [  ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-python-httpx) 



   

HTTPX is a new powerful HTTP client library for Python. It's quickly becoming the most popular option when it comes to HTTP connections in web scraping as it offers asynchronous client and http2 support.

In this highlight tutorial, we'll take a look at what makes Python's httpx so great for web scraping and how to use it effectively.

## Key Takeaways

Master Python web scraping with HTTPX library for modern HTTP/2 support, async requests, and advanced features like proxy rotation and session management.

- Use HTTPX for modern Python web scraping with HTTP/2 support and better performance than requests library
- Implement async web scraping with HTTPX for concurrent requests and improved scraping efficiency
- Handle proxy rotation and user agent management with HTTPX's built-in configuration options
- Use HTTPX's session management for cookie persistence and connection pooling in scraping workflows
- Apply proper timeout and retry logic with HTTPX for robust scraping applications
- Build scalable scrapers with HTTPX's async capabilities for high-performance data collection

**Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime.





## Installing httpx

HTTPX is a pure python package and so it can be easily installed using `pip` console command:

shell```shell
$ pip install httpx
```



Alternatively, it can be installed using poetry project package manager:

shell```shell
$ poetry init -d httpx
# or
$ poetry add httpx
```



## Using HTTPX

HTTPX can be used for individual requests directly and supports most of the popular HTTP functions like GET, POST requests and can unpack JSON responses as python dictionaries directly:

python```python
import httpx

# GET request
response = httpx.get("https://httpbin.dev/get")
print(response)
data = response.json()
print(data['url'])

# POST requests
payload = {"query": "foo"}
# application/json content:
response = httpx.post("https://httpbin.dev/post", json=payload)
# or formdata:
response = httpx.post("https://httpbin.dev/post", data=payload)
print(response)
data = response.json()
print(data['url'])
```



Here we used httpx for JSON loading using the `.json()` method of the response. Httpx comes with many convenient and accessible shortcuts like this making it a very accessible HTTP client for web scraping.

### Using httpx Client

For web scraping, it's best to use a `httpx.Client` which can apply custom settings like headers, cookies and proxies and configurations for the entire httpx session:

python```python
import httpx

with httpx.Client(
    # set headers for all requests
    headers={"x-secret": "foo"},
    # set cookies
    cookies={"language": "en"},
    # set one proxy for all HTTP and HTTPS requests
    proxy="http://222.1.1.1:8000",
) as session:
    response = session.get("https://httpbin.dev/get")
    print(response.status_code)

```



httpx client applies a set of configurations to all requests and even keeps track of cookies set by the server.

### Using httpx Asynchronously

To use httpx asynchronously with Python's `asyncio` the `httpx.AsyncClient()` object can be used:

python```python
import asyncio
import httpx

async def main():
    async with httpx.AsyncClient(
        # to limit asynchronous concurrent connections limits can be applied:
        limits=httpx.Limits(max_connections=10),
        # tip: increase timeouts for concurrent connections:
        timeout=httpx.Timeout(60.0),  # seconds
        # note: asyncClient takes in the same arguments like Client (like headers, cookies etc.)
    ) as client:
        # to make concurrent requests asyncio.gather can be used:
        urls = [
            "https://httpbin.dev/get",
            "https://httpbin.dev/get",
            "https://httpbin.dev/get",
        ]
        responses = asyncio.gather(*[client.get(url) for url in urls])
        # or asyncio.as_completed:
        for result in asyncio.as_completed([client.get(url) for url in urls]):
            response = await result
            print(response)

asyncio.run(main())
```



Note that when using `async with` all connections should finish before closing the `async with` statement otherwise exception will be raised:

python```python
RuntimeError: Cannot send a request, as the client has been closed.
```



Alternatively to the `async with` statement, the httpx AsyncClient can be opened/closed manually:

python```python
import asyncio
import httpx

async def main():
    client = httpx.AsyncClient()

    # do some scraping
    ...

    # close client
    await client.aclose()

asyncio.run(main())
```



## Troubleshooting HTTPX

While httpx for Python is a great library it's easy to run into some popular problems. Here are a few popular issues that can be encountered when web scraping with httpx and how to address them:

#### httpx.TimeoutException

The `httpx.TimeoutExcception` error occurs when a request takes longer than the specified/default timeout duration. Try raising the timeout parameter:

python```python
httpx.get("https://httpbin.dev/delay/10", timeout=httpx.Timeout(60.0))
```



#### httpx.ConnectError

The `httpx.ConnectError` exception is raised when connection issues are detected that can be caused by:

- unstable internet connection.
- server being unreachable.
- mistakes in the URL parameter.

#### httpx.TooManyRedirects

The `httpx.TooManyRedirects` exception is raised when automatic redirect following exceeds the maximum number of redirects.

To inspect a redirect manually instead, leave automatic following disabled:

python```python
response = httpx.get(
    "https://httpbin.dev/redirect/3",
    follow_redirects=False,  # do not follow the redirect
)
# then we can decide whether to follow the Location ourselves:
redirect_location = response.headers["Location"]
```



#### httpx.HTTPStatusError

The `httpx.HTTPStatusError` exception is raised by the response's `raise_for_status()` method when the status is outside the 200-299 range, such as 404:

python```python
response = httpx.get("https://httpbin.dev/status/404")
response.raise_for_status()
```



When web scraping status codes outside of 200-299 range can mean the scraper is being blocked.

#### httpx.UnsupportedProtocol

The `httpx.UnsupportedProtocol` error is raised when URL provided in the protocol is missing or is not part of `http://`, `https://`, `file://` or `ftp://`. This is most commonly encountered when URL is missing the `https://` part.

## Retrying HTTPX Requests

HTTPX transports can retry connection failures such as `ConnectError` and `ConnectTimeout`. Status-, response-content-, and broader exception-based retry policies need a tool such as [tenacity](https://pypi.org/project/tenacity/) (`pip install tenacity`). The corrected proxy, redirect, status, and retry examples in this guide were checked with HTTPX 0.28.1 and Tenacity 9.1.2 using local or mock transports. This is a focused check of those examples, not a full runtime validation of the tutorial.

Using `tenacity` we can retry selected 5xx status codes, HTTPX timeout and connection exceptions, and responses whose body contains a failure keyword:

python```python
import httpx
from tenacity import retry, stop_after_attempt, wait_fixed, retry_if_exception_type, retry_if_result

# Define the conditions for retrying based on HTTP status codes
def is_retryable_status_code(response):
    return response.status_code in [500, 502, 503, 504]

# Define the conditions for retrying based on response content
def is_retryable_content(response):
    return "you are blocked" in response.text.lower()

# Decorate the function with retry conditions and parameters
@retry(
    retry=(retry_if_exception_type((httpx.TimeoutException, httpx.ConnectError)) | retry_if_result(is_retryable_status_code) | retry_if_result(is_retryable_content)),
    stop=stop_after_attempt(3),
    wait=wait_fixed(5),
)
def fetch_url(url):
    try:
        # Return the response so result-based retry predicates can inspect it.
        return httpx.get(url)
    except httpx.RequestError as e:
        print(f"Request error: {e}")
        raise e

url = "https://httpbin.dev/get"
try:
    response = fetch_url(url)
    # Raise only after Tenacity has finished any result-based retries.
    response.raise_for_status()
    print(f"Successfully fetched URL: {url}")
    print(response.text)
except Exception as e:
    print(f"Failed to fetch URL: {url}")
    print(f"Error: {e}")
```



Above we are using tenacity's `retry` decorator and define our retrying rules for common httpx errors.

### Rotating Proxies for Retries

When it comes to handling blocking when web scraping with httpx proxy rotation can be used together with `tenacity` retry functionality.

In this example we'll take a look at a common web scraping pattern of rotating proxies and headers on scrape blocks. We'll add a retry that:

- Retries status codes 403 and 404
- Retries up to 5 times
- Sleeps randomly 1-5 seconds between retries
- Changes random proxy for each retry
- Changes random User-Agent request header for each retry

Using httpx and tenacity:

python```python
import httpx
import random
from tenacity import retry, stop_after_attempt, wait_random, retry_if_result
import asyncio


PROXY_POOL = [
    "http://2.56.119.93:5074",
    "http://185.199.229.156:7492",
    "http://185.199.228.220:7300",
    "http://185.199.231.45:8382",
    "http://188.74.210.207:6286",
    "http://188.74.183.10:8279",
    "http://188.74.210.21:6100",
    "http://45.155.68.129:8133",
    "http://154.95.36.199:6893",
    "http://45.94.47.66:8110",
]
USER_AGENT_POOL = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5",
]


# Define the conditions for retrying based on HTTP status codes
def is_retryable_status_code(response):
    return response.status_code in [403, 404]


# callback to modify scrape after each retry
def update_scrape_call(retry_state):
    # change to random proxy and User-Agent on each retry
    new_proxy = random.choice(PROXY_POOL)
    new_user_agent = random.choice(USER_AGENT_POOL)
    print(
        "retry {attempt_number}: {url} @ {proxy} with a new proxy {new_proxy}".format(
            attempt_number=retry_state.attempt_number,
            new_proxy=new_proxy,
            **retry_state.kwargs
        )
    )
    retry_state.kwargs["proxy"] = new_proxy
    retry_state.kwargs.setdefault("headers", {})["User-Agent"] = new_user_agent


@retry(
    # retry on bad status code
    retry=retry_if_result(is_retryable_status_code),
    # max 5 retries
    stop=stop_after_attempt(5),
    # wait randomly 1-5 seconds between retries
    wait=wait_random(min=1, max=5),
    # update scrape call on each retry
    before_sleep=update_scrape_call,
)
async def scrape(url, proxy, **client_kwargs):
    async with httpx.AsyncClient(
        proxy=proxy,
        **client_kwargs,
    ) as client:
        response = await client.get(url)
        return response
```



Above is a short demo of how to apply retry logic that is can rotate proxy and user agent string on each retry.

First, we define our proxy and user agent pools then use the `@retry` decorator to wrap our scrape function with tenacity's retry logic.

To modify each retry we are using `before_sleep` parameter which can update our scrape function call with new parameters on each retry.

Here's an example test run:

python```python
async def example_run():
    urls = [
        "https://httpbin.dev/ip",
        "https://httpbin.dev/ip",
        "https://httpbin.dev/ip",
        "https://httpbin.dev/status/403",
    ]
    to_scrape = [scrape(url=url, proxy=random.choice(PROXY_POOL), headers={"User-Agent": "foo"}) for url in urls]
    for result in asyncio.as_completed(to_scrape):
        response = await result
        print(response.json())


asyncio.run(example_run())
```



## Avoiding Blocking with Scrapfly

Scrapfly API offers a [Python SDK](https://scrapfly.io/docs/sdk/python) which is like httpx on steroids.



All functions of httpx are supported by [Scrapfly SDK](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) making migration effortless:

python```python
from scrapfly import ScrapeConfig, ScrapflyClient

client = ScrapflyClient(key="YOUR SCRAPFLY KEY")

result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev/get",
    # enable anti-scraping protection (like cloudflare or perimeterx) bypass
    asp=True,
    # select proxy country:
    country="US",
    # enable headless browser
    render_js=True,
))
print(result.content)

# tip: use concurrent scraping for blazing speeds:
to_scrape = [
    ScrapeConfig(url="https://httpbin.dev/get")
    for i in range(10)
]
async for result in client.concurrent_scrape(to_scrape):
    print(result.content)
```



Scrapfly SDK can be installed using pip console command and is free to try:

```
$ pip install scrapfly-sdk
```





## FAQ

HTTPX vs RequestsRequests is the most popular http client for Python known for being accessible and easy to work with. It's also an inspiration to HTTPX which is a natural successor to requests with modern python features like asyncio support and http2.







HTTPX vs AiohttpAiohttp is one of the first HTTP clients that supported asyncio and one of the inspirations of HTTPX. These two packages are very similar though aiohttp is more mature while httpx is newer but more feature rich. So, when it comes to aiohttp vs httpx the later is prefered in web scraping because of http2 support.







How to use HTTP2 with httpx?HTTP/2 support is optional in HTTPX. Install it first with `pip install 'httpx[http2]'`, then pass `http2=True` to `httpx.Client` or `httpx.AsyncClient`. The client uses HTTP/2 only when the server also supports it; otherwise it uses HTTP/1.1. Check `response.http_version` to see which protocol was negotiated.







How to automatically follow redirects in httpx?HTTPX doesn't follow redirects by default. To enable automatic redirect following, pass `follow_redirects=True` to request methods such as `httpx.get(url, follow_redirects=True)` or to a client such as `httpx.Client(follow_redirects=True)`.







How does httpx compare to the requests library for web scraping?HTTPX supports async requests and connection pooling by default. HTTP/2 is optional: install it with `pip install 'httpx[http2]'`, then enable it on a client with `http2=True`. The requests library is simpler but limited to synchronous calls. For request headers configuration, see our [Python requests headers guide](https://scrapfly.io/blog/posts/python-requests-headers-guide).









## Summary

HTTPX is a brilliant new HTTP client library that is quickly becoming the de facto standard in Python web scraping communities. It offers features like http2 and asyncio support which decreases the risk of blocking and allows concurrent web scraping.

Together with tenacity, httpx makes requesting web resources a breeze with powerful retry logic like proxy and user agent header rotation.



 

   [  Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents















 

  Table of Contents- [Key Takeaways](#key-takeaways)
- [Installing httpx](#installing-httpx)
- [Using HTTPX](#using-httpx)
- [Using httpx Client](#using-httpx-client)
- [Using httpx Asynchronously](#using-httpx-asynchronously)
- [Troubleshooting HTTPX](#troubleshooting-httpx)
- [Retrying HTTPX Requests](#retrying-httpx-requests)
- [Rotating Proxies for Retries](#rotating-proxies-for-retries)
- [Avoiding Blocking with Scrapfly](#avoiding-blocking-with-scrapfly)
- [FAQ](#faq)
- [Summary](#summary)
 
    Join the Newsletter  Get monthly web scraping insights 

 

  



Scale Your Web Scraping

Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits.

  No credit card required  1,000 free API credits  Anti-bot bypass included 

 [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) 

 Not ready? Get our newsletter instead. 

 

 ## Related Articles

 [  

 http python 

### How to Effectively Use User Agents for Web Scraping

In this article, we’ll take a look at the User-Agent header, what it is and how to use it in web scraping. We'll also ge...

 

 ](https://scrapfly.io/blog/posts/user-agent-header-in-web-scraping) [  

 crawling seo 

### What is Googlebot User Agent String?

Learn about Googlebot user agents, how to verify them, block unwanted crawlers, and optimize your site for better indexi...

 

 ](https://scrapfly.io/blog/posts/what-are-googlebot-user-agent-strings) [  

 http python 

### Web Scraping with Python

Introduction tutorial to web scraping with Python. How to collect and parse public data. Challenges, best practices and ...

 

 ](https://scrapfly.io/blog/posts/web-scraping-with-python) 

  ## Related Questions

- [ Q How to Set User Agent With cURL? ](https://scrapfly.io/blog/answers/how-to-set-curl-user-agent)
 
  



   



 Scale your web scraping effortlessly, **1,000 free credits** [Start Free](https://scrapfly.io/register)