What Python libraries support HTTP2?

Python has many different HTTP clients that can be used for web scraping. However, not all of them support HTTP2 which can be vital in avoiding web scraper blocking.

Here are the most popular HTTP clients that support HTTP2:

  • HTTPX - is one of the most popular new libraries for Python. HTTPX supports HTTP2 as well as asyncio making it great for web scraping:
import httpx
with httpx.Client(http2=True) as client:
    response = client.get("https://httpbin.dev/anything")
  • h2 is a low-level implementation of HTTP2 protocol. It's not recommended to use it directly for web scraping but it can be the only way to implement complex HTTP2 interactions for niche web scrapers.
  import h2.connection
  import h2.config

  config = h2.config.H2Configuration()
  conn = h2.connection.H2Connection(config=config)
  conn.send_headers(stream_id=stream_id, headers=headers)
  conn.send_data(stream_id, data)
  events = conn.receive_data(socket_data)

So, it's best to stick to httpx for HTTP2 though if you have a complex use case h2 can be adapted to extendible libraries like twisted.

How to Web Scrape with HTTPX and Python

For more on HTTPX in web scraping see our hands-on introduction article which covers everything you need to know when it comes to web scraping

How to Web Scrape with HTTPX and Python
Question tagged: Python, HTTP

Related Posts

How to Scrape Google SEO Keyword Data and Rankings

In this article, we’ll take a look at SEO web scraping, what it is and how to use it for better SEO keyword optimization. We’ll also create an SEO keyword scraper that scrapes Google search rankings and suggested keywords.

How to Effectively Use User Agents for Web Scraping

In this article, we’ll take a look at the User-Agent header, what it is and how to use it in web scraping. We'll also generate and rotate user agents to avoid web scraping blocking.

How to Observe E-Commerce Trends using Web Scraping

In this example web scraping project we'll be taking a look at monitoring E-Commerce trends using Python, web scraping and data visualization tools.