[Blog](https://scrapfly.io/blog)   /  [data-parsing](https://scrapfly.io/blog/tag/data-parsing)   /  [8 Top Python Web Scraping Libraries and APIs in 2026](https://scrapfly.io/blog/posts/top-10-web-scraping-libraries-in-python)   # 8 Top Python Web Scraping Libraries and APIs in 2026

 by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) Oct 22, 2026 20 min read [\#data-parsing](https://scrapfly.io/blog/tag/data-parsing) [\#python](https://scrapfly.io/blog/tag/python) [\#tools](https://scrapfly.io/blog/tag/tools) 

 [  ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Ftop-10-web-scraping-libraries-in-python "Share on LinkedIn") [  ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Ftop-10-web-scraping-libraries-in-python&text=8%20Top%20Python%20Web%20Scraping%20Libraries%20and%20APIs%20in%202026 "Share on X") [  ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Ftop-10-web-scraping-libraries-in-python "Share on Facebook")    

 

 

Summarize this article with

 [  ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Ftop-10-web-scraping-libraries-in-python) [  ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Ftop-10-web-scraping-libraries-in-python) [  ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Ftop-10-web-scraping-libraries-in-python) [  ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Ftop-10-web-scraping-libraries-in-python) [  ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Ftop-10-web-scraping-libraries-in-python) 



   

When a Python scraper comes back empty, the selectors aren't always to blame. Sometimes the page never loaded, and sometimes the data was in the first response, like the reviews on web-scraping.dev, which sit in the HTML as JSON.

But what about choosing the right Python library for each step? In this article, we'll go through the 8 best Python web scraping libraries and APIs, picking each one by what it returns. Let's dive in!



## Which Python Web Scraping Library or API Is Best in 2026?

Python is by far the most popular language used for web scraping. It's easy to learn, has a huge community and a massive ecosystem of libraries.

That being said, there's no single best library, as each one covers a different niche of the web scraping process, like:

- HTTP Connections
- Browser Automation
- Crawling
- HTML and JSON Parsing
- Data Quality and Validation

*we use many of these libraries in our [web scraping guide series](https://scrapfly.io/blog/tag/scrapeguide) if you want to see them in action*

Here are the versions we checked on PyPI, as of September 28, 2026:

| Package | Stable on 2026-09-28 | Release date | License | Python |
|---|---|---|---|---|
| `scrapfly-sdk` | 0.12.0 | Sep 8, 2026 | BSD-2-Clause | 3.6+ |
| `beautifulsoup4` | 4.15.0 | Jun 7, 2026 | MIT | 3.7+ |
| `Scrapy` | 2.19.0 | Sep 10, 2026 | BSD-3-Clause | 3.10+ |
| `playwright` | 1.63.0 | Sep 15, 2026 | Apache-2.0 | 3.10+ |
| `httpx` | 0.28.1 | Dec 6, 2024 | BSD-3-Clause | 3.8+ |
| `selenium` | 4.49.0 | Sep 9, 2026 | Apache-2.0 | 3.10+ |
| `jmespath` | 1.1.0 | Jan 22, 2026 | MIT | 3.9+ |
| `jsonpath-ng` | 1.8.0 | Feb 28, 2026 | Apache-2.0 | Not declared |
| `pydantic` | 2.13.5 | Aug 28, 2026 | MIT | 3.9+ |
| `Cerberus` | 1.3.8 | Nov 6, 2025 | ISC | 3.7+ |

Note that HTTPX hasn't had a release since 0.28.1 in December 2024, so for a new project, take a look at our [Python HTTP client comparison](https://scrapfly.io/blog/posts/best-python-http-clients) first.

Scrapfly is the only managed service on the list, so let's start with it, as it's the one that gets the page in the first place.



## 1. Scrapfly Web Scraping API: Best for Blocked and JavaScript-Heavy Pages

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale. Each product is equipped with an automatic bypass for any anti-bot system and we achieve this by:

- Maintaining a fleet of real, reinforced web browsers with real fingerprint profiles.
- Millions of self-healing proxies of the highest possible trust score.
- Constantly evolving and adapting to new anti-bot systems.

We've been doing this publicly since 2020 with the best bypass on the market!

For Python, there's the official [scrapfly-sdk](https://scrapfly.io/docs/sdk/python) package. All we have to do is create a `ScrapflyClient` with our API key and enable the bypass through the `unblocker` argument, which used to be called `asp`:

bash

```bash
pip install scrapfly-sdk==0.12.0 beautifulsoup4==4.15.0 lxml
```



python

```python
import os

from bs4 import BeautifulSoup
from scrapfly import ScrapeConfig, ScrapflyClient

client = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"])
result = client.scrape(
    ScrapeConfig(
        url="https://web-scraping.dev/products",
        unblocker=True,
        render_js=True,
        wait_for_selector=".product",
    )
)

soup = BeautifulSoup(result.content, "lxml")
for link in soup.select(".product h3 a"):
    print(link.text)
```



Here, `result.content` holds the rendered page HTML, so we can hand it straight to Beautiful Soup. We used version 0.12.0 of the SDK, from September 2026.

Note that the [API docs](https://scrapfly.io/docs/scrape-api/getting-started) ask for a 155-second read timeout, and the SDK already defaults to 160 seconds. Its source even has a `# 155 real` comment next to that number.

For pages that aren't protected, though, the open-source libraries below are enough, so let's start with Beautiful Soup.



## 2. Beautiful Soup: Best for Readable Static HTML Parsing

[Beautifulsoup](https://pypi.org/project/beautifulsoup4/) (aka bs4) is an HTML parser library in Python. Though it's much more than that.

Unlike LXML and Parsel, bs4 support parsing using accessible Pythonic methods like `find` and `find_all`:

python

```python
from bs4 import BeautifulSoup

html = """
<html>
  <title>My Title</title>
  <body>
    <div class="title">Product Title</div>
  </body>
</html>
"""

soup = BeautifulSoup(html, 'lxml')  # note: bs4 can use lxml under the hood which makes it super fast!
# find single element by node name
print(soup.find("title").text)
"My Title"
# find multiple using find_all and attribute matching
for element in soup.find_all("div", class_="title"):
    print(element.text)
```



This approach is more accessible and more readable than CSS or XPath selectors for beginners and often easier to maintain and develop when working with highly complex HTML documents.

BeautifulSoup4 also comes with many utility functions like HTML formatting and editing. For example:

python

```python
from bs4 import BeautifulSoup

html = """
<div><h1>The Avangers: </h1><a>End Game</a><p>is one of the most popular Marvel movies</p></div>
"""
soup = BeautifulSoup(html, "lxml")
soup.prettify()
"""
<html>
 <body>
  <div>
   <h1>
    The Avangers:
   </h1>
   <a>
    End Game
   </a>
   <p>
    is one of the most popular Marvel movies
   </p>
  </div>
 </body>
</html>
"""
```



As well as many other utility functions like HTML modification, selective parsing and cleanup.

We used `beautifulsoup4` 4.15.0 from June 2026, under the MIT license. Note that it's installed as `beautifulsoup4` but imported as `bs4`, and that it only parses the HTML we give it.

### When Should You Choose Parsel or lxml Instead of Beautiful Soup?

[LXML](https://pypi.org/project/lxml/) is a fast and feature-rich HTML/XML parser for Python. It's a wrapper around the C library `libxml2`, and it finds elements with [XPath selectors](https://scrapfly.io/blog/posts/parsing-html-with-xpath).

Let's use it on the same web-scraping.dev product page and pick up the product title, the price and the brand from the features table:

python

```python
import httpx
from lxml import html

response = httpx.get("https://web-scraping.dev/product/1")
tree = html.fromstring(response.text)

print(tree.xpath("//h3[contains(@class, 'product-title')]/text()"))
print(tree.xpath("//span[contains(@class, 'product-price')]/text()"))
# find the table cell next to the "brand" label:
print(tree.xpath("//td[text()='brand']/following-sibling::td/text()"))
```



Here is what we got from lxml 6.1.3:

text

```text
['Box of Chocolate Candy']
['$9.99 ', '$12.99']
['ChocoDelight']
```



We can see two prices here instead of one. That's because `contains()` also matched the `product-price-full` span, which holds the full $12.99 price, so XPath class matching needs a bit of care.

The brand line is where XPath shines, though. We found the `brand` cell by its text and took the next cell with `following-sibling`, and plain CSS selectors can't match an element by its text.

To take things even further, `lxml` is extended by [Parsel](https://pypi.org/project/parsel/) which simplifies the use API and adds [CSS selector](https://scrapfly.io/blog/posts/parsing-html-with-css) support. Here is the same page with Parsel 1.12.1:

python

```python
import httpx
from parsel import Selector

response = httpx.get("https://web-scraping.dev/product/1")
selector = Selector(response.text)

# use CSS selectors
print(selector.css(".product-title::text").get())
print(selector.css(".product-price::text").get())
print(selector.css(".variant::attr(data-variant-id)").getall())
# or XPath
print(selector.xpath("//td[text()='brand']/following-sibling::td/text()").get())
```



text

```text
Box of Chocolate Candy
$9.99 
['orange-small', 'orange-medium', 'orange-large', 'cherry-small', 'cherry-medium', 'cherry-large']
ChocoDelight
```



This time `.product-price` gave only the $9.99 price, as CSS class selectors match the whole class name. The `::text` and `::attr()` parts are Parsel extensions, and they pull the text or an attribute out right in the selector.

Beautiful Soup doesn't support these extensions at all. When I tried the same `.product-title::text` selector with its `select()` method, it crashed with this:

text

```text
NotImplementedError: Pseudo-element found at position 14
```



Scrapy's own selectors are built on Parsel too, and the [Scrapy docs](https://docs.scrapy.org/en/latest/topics/selectors.html) call them "a thin wrapper around parsel library". So, any selector we test with Parsel works in a Scrapy spider as it is.

Both lxml and Parsel are also a lot faster than Beautiful Soup. I parsed this product page 200 times with each one using Python's `timeit`, and here is the average per parse:

text

```text
lxml: 0.40 ms
parsel: 0.34 ms
bs4 + lxml: 3.77 ms
bs4 + html.parser: 4.91 ms
```



So, **for bigger scrapers, lxml or Parsel will save a lot of parsing time**, even when Beautiful Soup uses lxml underneath.

We used lxml 6.1.3 from September 2026 and Parsel 1.12.1 from September 28, 2026. Both are under the BSD-3-Clause license.

For more, see our [lxml guide](https://scrapfly.io/blog/posts/intro-to-parsing-html-xml-python-lxml) and [Parsel guide](https://scrapfly.io/blog/posts/guide-to-html-parsing-with-parsel-python). That being said, these parsers only work on HTML we already downloaded, so let's take a look at Scrapy for crawling whole websites!



## 3. Scrapy: Best for Complete Multi-Page Crawling Projects

When the job is crawling a whole website, [Scrapy](https://pypi.org/project/Scrapy/) is the framework to use.

Scrapy is a Python web scraping framework built around Twisted, an asynchronous networking engine. That being said, we don't have to interact with the underlying architecture, as Scrapy abstracts it away with its own interface.

Scrapy comes with an engine called Crawler. It handles the low-level logic, such as the HTTP connection, scheduling and the entire execution flow.

In simple terms, we must provide the Crawler with a Spider object to generate the requests, parse and retrieve the data to store:

python

```python
import scrapy


class ProductsSpider(scrapy.Spider):
    name = "products"
    allowed_domains = ["web-scraping.dev"]
    start_urls = ["https://web-scraping.dev/products"]

    def parse(self, response):
        product_urls = response.xpath("//div[@class='row product']/div/h3/a/@href").getall()
        yield from response.follow_all(product_urls, callback=self.parse_product)

    def parse_product(self, response):
        yield {
            "title": response.xpath("//h3[contains(@class, 'product-title')]/text()").get(),
            "price": response.xpath("//span[contains(@class, 'product-price')]/text()").get(),
        }
```



We can run it with `scrapy runspider products.py -o products.json`. Here is what we got in the crawl stats:

text

```text
 'downloader/response_status_count/200': 6,
 'finish_reason': 'finished',
 'item_scraped_count': 5,
```



The spider requested the products page and the five product pages it links to, and saved five items like `{'title': 'Blue Energy Potion', 'price': '$4.99'}`.

We used Scrapy 2.19.0 from September 2026, under the BSD-3-Clause license, and it needs Python 3.10 or newer. Note that Scrapy doesn't run JavaScript, so those pages need a browser or an API.

[Web Scraping With Scrapy: The Complete Guide in 2026Build and run a Scrapy 2.18 project with async start(), pagination, selector tests, item validation, pipelines, and JSON Lines export.](https://scrapfly.io/blog/posts/web-scraping-with-scrapy)



## 4. Playwright: Best for Modern JavaScript Pages and Network Inspection

[Headless browsers](https://scrapfly.io/blog/posts/scraping-using-browsers) are becoming very popular in web scraping as a way to deal with dynamic javascript and scraper blocking.

Many websites use complex front-end that generated data on demand either through background requests or javascript functions. To access this data we need a javascript execution environment and there's no better way than to employ a real headless web browser.

[Javascript fingerprinting](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-javascript) is also becoming a required step to scrape many modern websites.

So, to control a headless browser for web scraping in Python we have two popular libraries, Playwright and Selenium.

[Playwright](https://pypi.org/project/playwright/) is a modern take on browser automation offering all of these capabilities in modern asynchronous and synchronous APIs:

python

```python
from playwright.sync_api import sync_playwright

# Using synchronous Python:
with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context(viewport={'width': 1920, 'height': 1080})
    page = context.new_page()
    page.goto("https://web-scraping.dev/product/1")

    # wait for the element to be present in the DOM
    page.wait_for_selector('#reviews')

    # click the button
    page.wait_for_selector("#load-more-reviews", state="attached")
    page.click("#load-more-reviews")

    # print the HTML
    print(page.content())

    browser.close()

# or asynchronous
import asyncio
from playwright.async_api import async_playwright

async def run():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(viewport={'width': 1920, 'height': 1080})
        page = await context.new_page()
        await page.goto("https://web-scraping.dev/product/1")

        # wait for the element to be present in the DOM
        await page.wait_for_selector('#reviews')

        # click the button
        await page.wait_for_selector("#load-more-reviews", state="attached")
        await page.click("#load-more-reviews")

        # print the HTML
        print(await page.content())

        await browser.close()

asyncio.run(run())
```



We ran both versions with Playwright 1.63.0 from September 2026. Note that `pip install playwright` doesn't include the browser, and `playwright install chromium` downloaded a 114 MB Chrome Headless Shell 153 build for us.

Its `page.click()` also scrolled to the button and waited for it to be clickable, which saved us a workaround that Selenium needed.



## 5. HTTPX: Best for Existing Sync and Async Fetch Code

[HTTPX](https://pypi.org/project/httpx/) is a complete and modern HTTP client package for Python. It is inspired by the popular [requests](https://pypi.org/project/requests/) library but it takes things even further by supporting modern Python and HTTP features.

To start, it supports both asynchronous and synchronous Python APIs. This makes it easy to scale httpx-powered requests while also staying accessible in simple scripts:

python

```python
import httpx

# use Sync API
with httpx.Client() as client:
    response = client.get("https://web-scraping.dev/product/1")
    print(response.text)

# use Async API:
import asyncio

async def run():
    async with httpx.AsyncClient() as client:
        response = await client.get("https://web-scraping.dev/product/1")
        print(response.text)

asyncio.run(run())
```



HTTPX also supports HTTP/2 through the `httpx[http2]` extra. When I tested it on a TLS fingerprint checker, though, it had the same JA4 fingerprint with and without HTTP/2, `t13d1712h1_ab0a1bf427ad_8e6e362c5eac`.

Note that 0.28.1 from December 2024 is still its latest release. It works fine in existing code, but for a new project, see our [Python HTTP client comparison](https://scrapfly.io/blog/posts/best-python-http-clients).



Scrapfly

#### Extract structured data automatically?

Scrapfly's Extraction API uses AI to turn any webpage into structured data — no selectors needed.

[Try Free →](https://scrapfly.io/register)## 6. Selenium: Best for Existing WebDriver and Grid Infrastructure

[Selenium](https://pypi.org/project/selenium/) is one of the first major browser automation libraries and it can do almost anything a browser can do. It was first made for testing websites, and it drives the browser through the W3C WebDriver protocol.

Let's use it to scrape the reviews from the same web-scraping.dev product page. We'll wait for the reviews to load, click the Load More button and then print each review:

python

```python
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# configure browser
options = Options()
options.add_argument("--headless=new")
options.add_argument("--window-size=1920,1080")

driver = webdriver.Chrome(options=options)
driver.get("https://web-scraping.dev/product/1")
wait = WebDriverWait(driver=driver, timeout=5)
# wait for page to load by waiting for reviews to appear on the page
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "#reviews .review")))
# click buttons:
button = wait.until(EC.element_to_be_clickable((By.ID, "load-more-reviews")))
ActionChains(driver).move_to_element(button).click().perform()

# parse the rendered reviews:
for review in driver.find_elements(By.CSS_SELECTOR, "#reviews .review"):
    print(review.text)
driver.quit()
```



Here is what we got with Selenium 4.49.0 and Chrome 154:

text

```text
2022-07-22
Absolutely delicious! The orange flavor is my favorite.
2022-08-16
I bought these as a gift, and they were well received. Will definitely purchase again.
2022-09-10
Nice variety of flavors. The chocolate is rich and smooth.
2022-10-02
The cherry flavor is amazing. Will be buying more.
2022-11-05
A bit pricey, but the quality of the chocolate is worth it.
```



We can see all five reviews, but our first try printed nothing. That's because it waited for `#reviews` like the 2023 version of this example, and that div is in the HTML before any review is added to it.

The page's JavaScript adds the reviews half a second after it loads, with a `setTimeout(renderReviews, 500)` call. So, **it's better to wait for the elements that hold the data**, which is `#reviews .review` here.

Clicking the button didn't work right away either. With a plain `button.click()`, the script crashed with this error:

text

```text
selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element is not clickable at point (386, 2090)
```



Headless Chrome gave us a viewport only 937 pixels tall, even with `--window-size=1920,1080`, and the button was much lower on the page. Moving the mouse to it with `ActionChains` first scrolled it into view, and the click went through!

Another thing we noticed is the 2023 `options.headless = True` line. It doesn't exist in Selenium 4.49.0 anymore, and Python doesn't complain about it, but Chrome starts as a normal browser with a `Chrome/154.0.0.0` user agent.

With `--headless=new` it does run headless, but here is the `User-Agent` header that httpbin.dev got from it:

json

```json
{
  "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/154.0.0.0 Safari/537.36"
}
```



So, websites can see it's headless Chrome from the very first request.

Selenium Manager also downloaded chromedriver 154 for us on the first run, so there was no driver to install by hand like in older Selenium versions.

Compared to Playwright, Selenium took more work on this page, as Playwright's `page.click()` scrolled to the button by itself. We'd mostly use Selenium when a Selenium Grid or another WebDriver setup is already running.

That being said, the same five reviews are also in the page's plain HTML as JSON, inside a `<script id="reviews-data">` tag. For hidden data like this, we can skip the browser and query the JSON with JMESPath or JSONPath!



## 7. JMESPath and JSONPath: Best for Querying Hidden JSON

[JMESPath](https://pypi.org/project/jmespath/) and [JSONPath](https://pypi.org/project/jsonpath-ng/) are two libraries that allow you to query JSON data using a query language similar to XPath.

As JSON is becoming bigger and bigger in the web scraping space, these libraries are becoming essential.

For example, with JMESPath we can query and reshape and flatten JSON datasets with ease:

python

```python
import jmespath

data = {
  "people": [
    {
      "name": "Foo",
      "age": 33,
      "addresses": [
        "123 Main St", "California", "US"
      ],
      "primary_email": "foo@email.com",
      "secondary_email": "bar@email.com",
    },
  ]
}
result = jmespath.search("""
  people[].{
    first_name: name,
    age_in_years: age,
    address: addresses[0],
    state: addresses[1],
    country: addresses[2],
    emails: [primary_email, secondary_email]
  }
""", data)
print(result)
```



text

```text
[{'first_name': 'Foo', 'age_in_years': 33, 'address': '123 Main St', 'state': 'California', 'country': 'US', 'emails': ['foo@email.com', 'bar@email.com']}]
```



This feature is especially useful when scraping [hidden web data](https://scrapfly.io/blog/posts/how-to-scrape-hidden-web-data) which can result in *massive* JSON datasets that are hard to navigate and digest.

Alternatively, JSONPath focuses less on reshaping datasets and more on selecting values from complex, heavily nested JSON datasets and supports advanced matching functions:

python

```python
import jsonpath_ng.ext as jp

data = {
    "products": [
        {"name": "Apple", "price": 12.88, "tags": ["fruit", "red"]},
        {"name": "Peach", "price": 27.25, "tags": ["fruit", "yellow"]},
        {"name": "Cake", "tags": ["pastry", "sweet"]},
    ]
}

# find all product names:
query = jp.parse("products[*].name")
for match in query.find(data):
    print(match.value)

# find all products with price > 20
query = jp.parse("products[?price>20].name")
for match in query.find(data):
    print(match.value)
```



text

```text
Apple
Peach
Cake
Peach
```



The killer feature of JSONPath is the recursive `$..` selector which allows to quickly select values by key anywhere in the dataset (similar to the beloved XPath's `//` selector):

python

```python
import jsonpath_ng.ext as jp

data = {
    "products": [
        {"name": "Apple", "price": 12.88},
        {"name": "Peach", "price": 27.25},
        {"name": "Cake"},
        {"multiproduct": [{"name": "Carrot"}, {"name": "Pineapple"}]}
    ]
}

# find all "name" fields no matter where they are:
query = jp.parse("$..name")
for match in query.find(data):
    print(match.value)
```



text

```text
Apple
Peach
Cake
Carrot
Pineapple
```



Note that both examples import `jsonpath_ng.ext`. With the plain `jsonpath_ng.parse()`, the `?price>20` filter fails with `Unexpected character: ?`.

We used JMESPath 1.1.0 from January 2026 and jsonpath-ng 1.8.0 from February 2026.



## 8. Pydantic and Cerberus: Best for Validating Scraped Records

An often overlooked process of web scraping is the data quality assurance step. Web scraping is a unique niche where result datasets are highly dynamic making quality testing very difficult.

Fortunately, there are several tools that can help out with ensuring web scraped data quality.

For web scraping applications that require real-time data validation [Pydantic](https://pypi.org/project/pydantic/) is a great choice. Pydantic allows specifying strict data models that can be used to validate and morph scraped data:

python

```python
from datetime import date
from typing import Optional

from pydantic import BaseModel, field_validator

# example for scraped company data
class Company(BaseModel):
    # define allowed field names and types:
    size: int
    founded: int
    revenue_currency: str
    hq_city: str
    hq_state: Optional[str] = None  # some fields can be optional (i.e. have value of None)

    # then we can define any custom validation functions:
    @field_validator("size")
    @classmethod
    def must_be_reasonable_size(cls, v):
        if not (0 < v < 20_000):
            raise ValueError(f"unreasonable company size: {v}")
        return v

    @field_validator("founded")
    @classmethod
    def must_be_reasonable_year(cls, v):
        if not (1900 < v <= date.today().year):
            raise ValueError(f"unreasonable found date: {v}")
        return v

    @field_validator("hq_state")
    @classmethod
    def looks_like_state(cls, v):
        if v is not None and len(v) != 2:
            raise ValueError(f'state should be 2 character long, got "{v}"')
        return v


Company(size=50, founded=2031, revenue_currency="USD", hq_city="Austin", hq_state="Texas")
```



Here is what we got.

text

```text
pydantic_core._pydantic_core.ValidationError: 2 validation errors for Company
founded
  Value error, unreasonable found date: 2031 [type=value_error, input_value=2031, input_type=int]
    For further information visit https://errors.pydantic.dev/2.13/v/value_error
hq_state
  Value error, state should be 2 character long, got "Texas" [type=value_error, input_value='Texas', input_type=str]
    For further information visit https://errors.pydantic.dev/2.13/v/value_error
```



This is the Pydantic 2 version of the model, with `field_validator` and an explicit `None` default for the optional field. We ran it with Pydantic 2.13.5 from August 2026.

For data web scrapers that require more flexibility and less strict data validation [Cerberus](https://pypi.org/project/Cerberus/) is a great choice. Cerberus is a schema validation library that allows specifying data models using a simple dictionary syntax:

python

```python
from cerberus import Validator


def validate_name(field, value, error):
    """function for validating"""
    if "." in value:
        error(field, f"contains a dot character: {value}")
    if value.lower() in ["classified", "redacted", "missing"]:
        error(field, f"redacted value: {value}")
    if "<" in value.lower() and ">" in value.lower():
        error(field, f"contains html nodes: {value}")


schema = {
    "name": {
        # name should be a string
        "type": "string",
        # between 2 and 20 characters
        "minlength": 2,
        "maxlength": 20,
        # extra validation
        "check_with": validate_name,
    },
}

v = Validator(schema)

v.validate({"name": "H."})
print(v.errors)
# {'name': ['contains a dot character: H.']}
```



The killer feature of Cerberus is the ease of use and ability to define flexible schemas that can work with highly dynamic web-scraped data. At Scrapfly, we use Cerberus to test [our example scrapers](https://github.com/scrapfly/scrapfly-scrapers).



## How Do You Choose a Python Web Scraping Stack by Job?

A [Reddit thread on r/webscraping](https://www.reddit.com/r/webscraping/comments/1i0hthe/what_are_the_current_best_python_libs_for_web/) asked "What are the current best Python libs for Web Scraping and why?" The poster was "Currently working with Selenium + Beautiful Soup, but heard about Scrapy and Playwright".

We usually answer that by sending one plain request to the page first. A block page sends us to Scrapfly, and missing data to a browser, so let's see what the web-scraping.dev product page from the Selenium section gives us:

python

```python
import httpx
from bs4 import BeautifulSoup

response = httpx.get("https://web-scraping.dev/product/1")
soup = BeautifulSoup(response.text, "lxml")

print(response.status_code)
print(len(soup.select("#reviews .review")))
print("Absolutely delicious!" in response.text)
```



text

```text
200
0
True
```



We got a 200 response but zero review elements, as the page renders them with JavaScript. The first review's text is still in the response, though, so the reviews have to be somewhere in the HTML!

The text shows up twice, once in the page's JSON-LD `<script>` and once in a `<script id="reviews-data">` tag. So, for this page **we don't need a browser at all**, and HTTPX, Beautiful Soup, JMESPath and Pydantic are enough:

bash

```bash
pip install httpx==0.28.1 beautifulsoup4==4.15.0 jmespath==1.1.0 pydantic==2.13.5 lxml
```



python

```python
import json

import httpx
import jmespath
from bs4 import BeautifulSoup
from pydantic import BaseModel, Field


class Review(BaseModel):
    review_id: str
    rating: int = Field(ge=1, le=5)


response = httpx.get("https://web-scraping.dev/product/1")
soup = BeautifulSoup(response.text, "lxml")
reviews = json.loads(soup.select_one("#reviews-data").text)

five_star = jmespath.search(
    "[?rating == `5`].{review_id: id, rating: rating}", reviews
)
records = [Review.model_validate(item) for item in five_star]

print("status:", response.status_code)
print("reviews:", len(reviews))
print("five-star reviews:", len(records))
print("first:", records[0].review_id)
```



Here is what we got.

text

```text
status: 200
reviews: 5
five-star reviews: 3
first: chocolate-candy-box-1
```



We can see that the page has five reviews, and three of them are five-star ones, which all passed our `Review` model.

To wrap up this article, let's take a look at some frequently asked questions.



## FAQ

Can Beautiful Soup Work With Scrapfly or Scrapy?Yes. Beautiful Soup can parse any HTML string, so it works with Scrapfly's `result.content` and with `response.text` inside a Scrapy callback.







Does HTTP/2 Prevent Python Scrapers From Being Blocked?No. In our test, HTTPX sent the same TLS fingerprint with and without HTTP/2, and websites also look at IP addresses, headers, cookies and JavaScript. For more, see our [guide to scraping without getting blocked](https://scrapfly.io/blog/posts/how-to-bypass-anti-bot-protection-when-web-scraping).







Is HTTPX Still Usable for Web Scraping in 2026?Yes. It works fine, though its latest release is still 0.28.1 from December 2024, so for new projects, compare it with the other clients first.







When Should Python Scrapers Use JMESPath Instead of JSONPath?JMESPath is better for reshaping JSON into new structures, while JSONPath is better for finding values deep inside nested datasets, like with its recursive `$..` selector.









## Conclusion: Which Python Web Scraping Stack Should You Use?

In this article, we've covered the top 8 Python packages and APIs for web scraping that cover several steps of the web scraping process.

For web scraping connections, we've covered `httpx` which is a brilliant, feature-rich HTTP client. On the other hand, there's `selenium` and `playwright` which use a whole headless web browser.

For data parsing, we've taken a look at the three most popular HTML parsing tools, `lxml`, `parsel` and `beautifulsoup`. For JSON parsing `jmespath` and `jsonpath` which work great in web scraping environments.

For crawling whole websites, there's `scrapy`, and for the final step of a web scraping process, we've looked at two different data validation tools, strict `pydantic` and the flexible `cerberus`.

Finally, when blocking or JavaScript makes getting the page the hard part, the [Scrapfly Web Scraping API](https://scrapfly.io/products/web-scraping-api) can fetch it for us. Check out Scrapfly for free!



 

   [  Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents















 

  Table of Contents

- [Which Python Web Scraping Library or API Is Best in 2026?](#which-python-web-scraping-library-or-api-is-best-in-2026)
- [1. Scrapfly Web Scraping API: Best for Blocked and JavaScript-Heavy Pages](#1-scrapfly-web-scraping-api-best-for-blocked-and-javascript-heavy-pages)
- [2. Beautiful Soup: Best for Readable Static HTML Parsing](#2-beautiful-soup-best-for-readable-static-html-parsing)
- [When Should You Choose Parsel or lxml Instead of Beautiful Soup?](#when-should-you-choose-parsel-or-lxml-instead-of-beautiful-soup)
- [3. Scrapy: Best for Complete Multi-Page Crawling Projects](#3-scrapy-best-for-complete-multi-page-crawling-projects)
- [4. Playwright: Best for Modern JavaScript Pages and Network Inspection](#4-playwright-best-for-modern-javascript-pages-and-network-inspection)
- [5. HTTPX: Best for Existing Sync and Async Fetch Code](#5-httpx-best-for-existing-sync-and-async-fetch-code)
- [6. Selenium: Best for Existing WebDriver and Grid Infrastructure](#6-selenium-best-for-existing-webdriver-and-grid-infrastructure)
- [7. JMESPath and JSONPath: Best for Querying Hidden JSON](#7-jmespath-and-jsonpath-best-for-querying-hidden-json)
- [8. Pydantic and Cerberus: Best for Validating Scraped Records](#8-pydantic-and-cerberus-best-for-validating-scraped-records)
- [How Do You Choose a Python Web Scraping Stack by Job?](#how-do-you-choose-a-python-web-scraping-stack-by-job)
- [FAQ](#faq)
- [Conclusion: Which Python Web Scraping Stack Should You Use?](#conclusion-which-python-web-scraping-stack-should-you-use)
 
    Join the Newsletter  Get monthly web scraping insights 

 

  



Scale Your Web Scraping

Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits.

  No credit card required  1,000 free API credits  Anti-bot bypass included 

 [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) 

 Not ready? Get our newsletter instead. 

 

 ## Related Articles

 [     

 python scrapeguide 

### How to Scrape Google Jobs in 2026

Learn how to scrape Google Jobs with Python and Scrapfly. Build a job-search URL, render and scroll the results, and ext...

 

 ](https://scrapfly.io/blog/posts/how-to-scrape-google-jobs) [  

 http python 

### Web Scraping with Python

Introduction tutorial to web scraping with Python. How to collect and parse public data. Challenges, best practices and ...

 

 ](https://scrapfly.io/blog/posts/web-scraping-with-python) [  

 python 

### How to Crawl the Web with Python

Introduction to web crawling with Python. What is web crawling? How it differs from web scraping? And a deep dive into c...

 

 ](https://scrapfly.io/blog/posts/crawling-with-python) 

  



   



 Extract structured data with AI, **1,000 free credits** [Start Free](https://scrapfly.io/register)