How to use proxies with Python httpx?

by scrapecrow May 16, 2023

Python's httpx HTTP client package supports both HTTP and SOCKS5 proxies. Here's how to use proxies with httpx:

import httpx
from urllib.parse import quote

# proxy pattern is:
# scheme://username:password@IP:PORT
# For example:
# no auth HTTP proxy:
my_proxy = "http://160.11.12.13:1020"
# or socks5
my_proxy = "http://160.11.12.13:1020|socks5"
# proxy with authentication
my_proxy = "http://my_username:my_password@160.11.12.13:1020"
# note: that username and password should be url quoted if they contain URL sensitive characters like "@":
my_proxy = f"http://{quote('foo@bar.com')}:{quote('password@123')}@160.11.12.13:1020"

proxies = {
    # this proxy will be applied to all http:// urls
    'http://': 'http://160.11.12.13:1020',
    # this proxy will be applied to all https:// urls (not the S)
    'https://': 'http://160.11.12.13:1020',
    # we can also use proxy only for specific pages
    'https://httpbin.dev': 'http://160.11.12.13:1020',
}

with httpx.Client(proxies=proxies) as client:
    r = client.get("https://httpbin.dev/ip")
# or async
async with httpx.AsyncClient(proxies=proxies) as client:
    r = await client.get("https://httpbin.dev/ip")

Note that proxy can also be set through the standard *_PROXY environment variables:

$ export HTTP_PROXY="http://160.11.12.13:1020"
$ export HTTPS_PROXY="http://160.11.12.13:1020"
$ export ALL_PROXY="socks://160.11.12.13:1020"
$ python
import httpx
# this will use the proxies we set
with httpx.Client() as client:
    r = client.get("https://httpbin.dev/ip")

When web scraping, it's best to rotate proxies for each request. For that see our article: How to Rotate Proxies in Web Scraping

How to Web Scrape with HTTPX and Python

Intro to using Python's httpx library for web scraping. Proxy and user agent rotation and common web scraping challenges, tips and tricks.

How to Web Scrape with HTTPX and Python

Related Articles

Guide to Python requests POST method

Discover how to use Python's requests library for POST requests, including JSON, form data, and file uploads, along with response handling tips.

PYTHON
REQUESTS
HTTP
Guide to Python requests POST method

Guide to Python Requests Headers

Our guide to request headers for Python requests library. How to configure and what do they mean.

PYTHON
REQUESTS
HTTP
Guide to Python Requests Headers

How to Scrape Forms

Learn how to scrape forms through a step-by-step guide using HTTP clients and headless browsers.

HEADLESS-BROWSER
PYTHON
HTTPX
INTRO
NODEJS
How to Scrape Forms

FlareSolverr Guide: Bypass Cloudflare While Scraping

In this article, we'll explore the FlareSolverr tool and how to use it to get around Cloudflare while scraping. We'll start by explaining what FlareSolverr is, how it works, how to install and use it. Let's get started!

PYTHON
TOOLS
BLOCKING
HTTP
FlareSolverr Guide: Bypass Cloudflare While Scraping

How to Effectively Use User Agents for Web Scraping

In this article, we’ll take a look at the User-Agent header, what it is and how to use it in web scraping. We'll also generate and rotate user agents to avoid web scraping blocking.

HTTP
PYTHON
How to Effectively Use User Agents for Web Scraping

How to Scrape in Another Language, Currency or Location

Localization allows for adapting websites content by changing language and currency. So, how do we scrape it? We'll take a look at the most common methods for changing language, currency and other locality details in web scraping.

PYTHON
HEADLESS-BROWSER
HTTP
How to Scrape in Another Language, Currency or Location