     [Answers](https://scrapfly.io/blog)   /  [beautifulsoup](https://scrapfly.io/blog/tag/beautifulsoup)   /  [Scrapy vs BeautifulSoup: Which Should You Use?](https://scrapfly.io/blog/answers/scrapy-vs-beautifulsoup)   

# Scrapy vs BeautifulSoup: Which Should You Use?

 by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) Nov 17, 2026 22 min read [\#beautifulsoup](https://scrapfly.io/blog/tag/beautifulsoup) [\#scrapy](https://scrapfly.io/blog/tag/scrapy) 

 [  ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-vs-beautifulsoup "Share on LinkedIn") [  ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-vs-beautifulsoup&text=Scrapy%20vs%20BeautifulSoup%3A%20Which%20Should%20You%20Use%3F "Share on X") [  ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-vs-beautifulsoup "Share on Facebook")    

 

 

Summarize this article with

 [  ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-vs-beautifulsoup) [  ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-vs-beautifulsoup) [  ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-vs-beautifulsoup) [  ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-vs-beautifulsoup) [  ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-vs-beautifulsoup) 



Scrapy and Beautiful Soup solve different parts of the same scraping job. Beautiful Soup parses HTML you already fetched. Scrapy runs the crawl: it schedules requests, follows links, retries eligible failures, and yields records through one framework. Use Beautiful Soup with an HTTP client for a small direct script, Scrapy for a maintained multi-page crawl, or both when you want Scrapy's crawl lifecycle with Beautiful Soup's parsing API.



## Key Takeaways

Scrapy is an HTTP-based web crawling framework with built-in tools for common web scraping utilities, such as middlewares, runners and parsers. However, the biggest advantage of Scrapy lies in its non-blocking networking infrastructure that makes requesting web pages more efficient.

- For HTML parsing, there's `beautifulsoup4` which needs to be paired with an HTTP client like `requests` or `httpx` to retrieve web pages.
- For known URLs, `requests` and `BeautifulSoup` are all we need, while Scrapy is best when the crawl has to follow links and run again.
- For existing `BeautifulSoup` code, we can pass `response.text` to `BeautifulSoup` inside a Scrapy spider callback instead of rewriting selectors in `parsel`.
- Finally, for web scraping speed, IO-bound HTTP requests take much more time than CPU-bound HTML parsing with `lxml`.

**Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime.







## Scrapy vs Beautiful Soup at a glance

Here's the short version, with Requests doing the fetching for Beautiful Soup:

| Decision axis | Requests + Beautiful Soup | Scrapy |
|---|---|---|
| Primary role | Fetch responses explicitly, then parse the HTML/XML | Crawl, schedule requests, and extract records |
| Fetches pages | Requests or another HTTP client does | Scrapy's downloader does |
| Parsing API | `find`, `find_all`, `select`, tree navigation | Parsel selectors through `response.css()` and `response.xpath()` |
| CSS / XPath | CSS through Beautiful Soup, no native XPath API | CSS and XPath |
| URL queue | We write and maintain it | The scheduler manages the requests we yield |
| Concurrency | Depends on the HTTP client and the orchestration we add | Built into the crawl engine and settings |
| Retries and cookies | We choose and wire the supporting code | Framework middleware provides lifecycle points and defaults |
| Record processing | Our own functions and storage code | Item pipelines and feed exports |
| Smallest shape | One Python script | One-file spider with `runspider`, or a full project |
| Best fit | Direct, exploratory, or disposable scripts | Recurring multi-page crawls that must remain maintainable |

Beautifulsoup is a python library that is used for parsing HTML documents. Using it we can navigate HTML data to extract/delete/replace particular HTML elements. It also comes with utility functions like visual formatting and parse tree cleanup.

It's used for parsing collected HTML data and it's really good at it. It doesn't collect anything itself, so we'll also be using `requests` package in our example to download the web content.

Unlike LXML and Parsel, bs4 support parsing using accessible Pythonic methods like `find` and `find_all`. There's also `select` and `select_one` for CSS selectors, and `.parent` when we need to step up the tree.

This approach is more accessible and more readable than CSS or XPath selectors for beginners and often easier to maintain and develop when working with highly complex HTML documents.

XPath is where bs4 stops. Here's what we got when we tried it on the products page anyway:

python

```python
soup.xpath("//h3")
# TypeError: 'NoneType' object is not callable
```



Bs4 reads `.xpath` as a search for an `<xpath>` tag, finds none and hands back `None`.

Scrapy, on the other hand, is a full framework. Scrapy comes with its own crawl engine. It handles the low-level logic, such as the HTTP connection, scheduling and the entire execution flow.

In simple terms, we must provide Scrapy with a Spider object to generate the requests, parse and retrieve the data to store.

That being said, we don't have to interact with the underlying architecture. Scrapy abstracts it away with its own interface. From the development perspective, we'll mostly deal with the equivalent logic in callbacks and generators.

For parsing, Scrapy doesn't use bs4. To take things even further, `lxml` is extended by Parsel which simplifies the use API and adds CSS selector support. That's what `response.css()` and `response.xpath()` are.

Parsel is becoming the de facto way to parse large sets of HTML documents as it combines the speed of `lxml` with the ease of use of CSS and XPath selectors.

Many CSS selector clients expand functionality with their own extra syntax. For example, for selecting node attributes or inner text extra pseudo classes like `::attr`, `::text` are being added as extra functionality. Hence `h3 a::text` in the spider below.

Scrapy is an asynchronous framework. Therefore, most of the actions are executed in the background, which allows for highly concurrent and efficient logic.

In this context, a callback is a function that's attached to a background task, which called upon the successful finish of this task. We'll write generators that generate either requests with callbacks or results that will be saved to storage.

Repeated requests don't get through either. Here's what the log said when we yielded the second page twice:

python

```python
yield response.follow("/products?page=2")
yield response.follow("/products?page=2")
# DEBUG: Filtered duplicate request: <GET https://web-scraping.dev/products?page=2>
```



Only one of them was downloaded.

Scrapy offers various convenient interception points for different actions performed by the web scraping engine. For example, downloader middleware allows for the pre-processing of outgoing requests and the post-processing of incoming responses.

We can use this to design custom connection logic, such as retrying requests, dropping others or implementing connection caching. Retries and cookies are already on by default.

Pipelines are essentially data post-processors. Whenever our spider generates some results, they are piped through registered pipelines, and the final output is sent to our feed (be it a log or a feed export).

On the Requests side we get a `requests.Session`. In other words, a session will establish the connection only *once* and continue exchanging our requests until we close it.

Session client not only makes the connection more efficient but provides many convenient features like global headers settings, automatic cookie management and so on. Retries and the URL queue are still our own code.

Requests is the most popular http client for Python known for being accessible and easy to work with. It's also an inspiration to HTTPX which is a natural successor to requests with modern python features like asyncio support and http2.

To start, it supports both asynchronous and synchronous Python APIs. This makes it easy to scale httpx-powered requests while also staying accessible in simple scripts.

Scrapy comes with many batteries-included features like bad response retrying and efficient request scheduling. However, Scrapy - being a full framework - can be difficult to patch and integrate with other python technologies.

A whole project isn't needed for one spider though. This is what `scrapy -h` lists:

```
runspider     Run a spider from a Python file, no project required
```



That's the command our example uses below.

Let's take a look at why they're still not direct equivalents.



## Why Scrapy and Beautiful Soup are not direct equivalents

The [Scrapy FAQ](https://docs.scrapy.org/en/latest/faq.html#how-does-scrapy-compare-to-beautifulsoup-or-lxml) answers this one itself. Here's the entry as it reads today:

> BeautifulSoup and lxml are libraries for parsing HTML and XML. Scrapy is an application framework for writing web spiders that crawl web sites and extract data from them.
> 
> Scrapy provides a built-in mechanism for extracting data (called selectors) but you can easily use BeautifulSoup (or lxml) instead, if you feel more comfortable working with them. After all, they're just parsing libraries which can be imported and used from any Python code.
> 
> In other words, comparing BeautifulSoup (or lxml) to Scrapy is like comparing jinja2 to Django.

So the real matchup is Scrapy against Requests, Beautiful Soup and the queue and retry code we'd write around them.

Those built-in selectors are Parsel. `.get()` gives us the first match and `.getall()` gives all of them. Here's what they returned on the products page:

python

```python
sel.css(".product h3 a::text").get()
# 'Box of Chocolate Candy'
sel.css(".product h3 a::text").getall()
# ['Box of Chocolate Candy', 'Dark Red Energy Potion', 'Teal Energy Potion', 'Red Energy Potion', 'Blue Energy Potion']
```



Beautiful Soup has one more choice to make, the parser backend. Bs4 is pretty big and comes with several backends that provide HTML parsing algorithms that differ very slightly:

- [html.parser](https://docs.python.org/3/library/html.parser.html) - python's built-in parser, which is written in python meaning it's always available though it's a bit slower.
- [lxml](https://lxml.de/) - C-based library for HTML parsing: very fast, but can be a bit more difficult to install.
- [html5lib](https://github.com/html5lib/html5lib-python) - another parser written in python that is intended to be fully html5 compliant.

The [Beautiful Soup docs](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#differences-between-parsers) say when that "very slightly" stops being true:

> There are also differences between HTML parsers. If you give Beautiful Soup a perfectly-formed HTML document, these differences won't matter. One parser will be faster than another, but they'll all give you a data structure that looks exactly like the original HTML document.
> 
> But if the document is not perfectly-formed, different parsers will give different results.

We ran their `<a></p>` example through all three:

python

```python
from bs4 import BeautifulSoup

for parser in ["html.parser", "lxml", "html5lib"]:
    print(parser, BeautifulSoup("<a></p>", parser))
```



```
html.parser <a></a>
lxml <html><body><a></a></body></html>
html5lib <html><head></head><body><a><p></p></a></body></html>
```



To summarize, it's best to stick with `lxml` backend because it's much faster, however `html.parser` is still a good option for smaller projects.

As for `html5lib` it's mostly best for edge cases where html5 specification compliance is necessary.

Now let's put both on the same listing page and compare the code.



## Scrapy vs Beautiful Soup: same page, same output

Before we go further, let's see a quick example of what both are capable of on the same page, [web-scraping.dev/products](https://web-scraping.dev/products).

In this article, we'll be using Python 3.13 with Scrapy 2.19.0, beautifulsoup4 4.15.0, lxml 6.1.3 and requests 2.34.2. All of these can be installed through the `pip install` console command:

shell

```shell
pip install scrapy beautifulsoup4 lxml requests
```



### Requests + Beautiful Soup

python

```python
import requests
from bs4 import BeautifulSoup

url = "https://web-scraping.dev/products"
response = requests.get(url, timeout=30)
response.raise_for_status()
soup = BeautifulSoup(response.text, "lxml")

products = []
for card in soup.select(".product")[:3]:
    products.append({
        "title": card.select_one("h3 a").get_text(strip=True),
        "price": card.select_one(".price").get_text(strip=True),
    })

print(products)
```



Here, we used a few clever CSS selectors to select the desired fields on the HTML. Here is a sample output to the results we got:

python

```python
[{'title': 'Box of Chocolate Candy', 'price': '24.99'}, {'title': 'Dark Red Energy Potion', 'price': '4.99'}, {'title': 'Teal Energy Potion', 'price': '4.99'}]
```



This example illustrates how easily we can parse web pages for product data and a few key features of beautifulsoup4.

### Scrapy in one file

python

```python
import scrapy


class ProductsSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://web-scraping.dev/products"]

    def parse(self, response):
        for card in response.css(".product")[:3]:
            yield {
                "title": card.css("h3 a::text").get().strip(),
                "price": card.css(".price::text").get().strip(),
            }
```



Let's break down the above idioms:

- `name` is used as a reference to this spider for `scrapy` commands such as `scrapy crawl <name>`, which would run this scraper.
- `start_urls` indicates the spider starting point, where `parse()` is the first callback. It represents a callback to execute the following instructions.

There are two ways to run Scrapy spiders: either through the `scrapy` command or by explicitly calling Scrapy via a Python script.

It is generally recommended to use the Scrapy CLI tool, as Scrapy is a complex system, and it is safer to provide it with a dedicated Python process.

Ours is a single file, so it's [`runspider`](https://docs.scrapy.org/en/latest/topics/commands.html#std-command-runspider) instead of `crawl`:

shell

```shell
scrapy runspider products_spider.py
...
2026-11-04 15:39:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://web-scraping.dev/products> (referer: None)
2026-11-04 15:39:44 [scrapy.core.scraper] DEBUG: Scraped from <200 https://web-scraping.dev/products>
{'title': 'Box of Chocolate Candy', 'price': '24.99'}
2026-11-04 15:39:44 [scrapy.core.scraper] DEBUG: Scraped from <200 https://web-scraping.dev/products>
{'title': 'Dark Red Energy Potion', 'price': '4.99'}
2026-11-04 15:39:44 [scrapy.core.scraper] DEBUG: Scraped from <200 https://web-scraping.dev/products>
{'title': 'Teal Energy Potion', 'price': '4.99'}
```



Scrapy provides brilliant logs that log everything that the Scrapy engine is performing, as well as logging the returned results.

Moreover, Scrapy attaches some useful scrape statistics, such as how many items were scraped, how long it took for our scraper to finish, etc.

We have a spider that scrapes product data successfully and logs the results. To save to a file, we can update our `scrapy runspider` command with an output flag:

shell

```shell
scrapy runspider products_spider.py -O products.json
...
2026-11-04 15:39:45 [scrapy.extensions.feedexport] INFO: Stored json feed (3 items) in: products.json
```



And here's the `products.json` file it wrote:

json

```json
[
{"title": "Box of Chocolate Candy", "price": "24.99"},
{"title": "Dark Red Energy Potion", "price": "4.99"},
{"title": "Teal Energy Potion", "price": "4.99"}
]
```



Same three products either way. But what about more than one page?



## What changes when Beautiful Soup becomes a Scrapy crawl?

The crawler starts with a pool of URLs (the initial seed is often called *start urls*) and scrapes their responses (HTML data).

Responses are parsed for more URLs to follow which are being filtered and added to the next crawl loop pool.

This loop is repeated until no new URLs are found or a certain end condition is met like crawling depth or collected URL count is reached.

With Requests and Beautiful Soup, that whole loop is our code. Here's who owns each part once the job grows past one page:

| Requirement | Requests + Beautiful Soup | Scrapy |
|---|---|---|
| Follow pagination | We write the loop or queue | We yield requests, and the scheduler handles the queue |
| Concurrent downloads | We add an async client, worker pool, or executor | Built into the engine, tuned with the `CONCURRENT_REQUESTS` settings |
| Retry transient failures | We add a retry policy around the client | `RetryMiddleware`, 2 retries by default, tuned in settings |
| Preserve cookies | We use a client session | `CookiesMiddleware` tracks cookies unless disabled |
| Filter repeated requests | We track URLs or fingerprints ourselves | The scheduler's duplicate filter, skipped by `dont_filter=True` requests |
| Validate and transform records | We write functions and storage steps | Item pipelines, or values transformed in the callback before yielding |
| Export JSON/CSV | We write the output code | Feed exports such as `-O products.json` |
| Maintain several scrapers | We organize our own package and config | A project shares settings, middleware, pipelines, and commands |

To see it, we let our spider follow every pagination link on all 5 listing pages:

python

```python
import scrapy


class ProductsSpider(scrapy.Spider):
    name = "products"

    async def start(self):
        # start_urls skip the duplicate filter
        yield scrapy.Request("https://web-scraping.dev/products?page=1")

    def parse(self, response):
        for card in response.css(".product"):
            yield {
                "title": card.css("h3 a::text").get().strip(),
                "price": card.css(".price::text").get().strip(),
            }
        yield from response.follow_all(css=".paging a")
```



Here's what we got from `scrapy runspider products_spider.py -O products.json`:

```
2026-11-12 19:43:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://web-scraping.dev/products?page=1> (referer: None)
2026-11-12 19:43:46 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://web-scraping.dev/products?page=1> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2026-11-12 19:43:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://web-scraping.dev/products?page=5> (referer: https://web-scraping.dev/products?page=1)
2026-11-12 19:43:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://web-scraping.dev/products?page=2> (referer: https://web-scraping.dev/products?page=1)
2026-11-12 19:43:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://web-scraping.dev/products?page=4> (referer: https://web-scraping.dev/products?page=1)
2026-11-12 19:43:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://web-scraping.dev/products?page=3> (referer: https://web-scraping.dev/products?page=1)
2026-11-12 19:43:47 [scrapy.extensions.feedexport] INFO: Stored json feed (25 items) in: products.json
...
 'dupefilter/filtered': 25,
 'item_scraped_count': 25,
 'scheduler/enqueued': 5,
```



Pages 2 to 5 came back out of order, and none of the 25 repeated links reached the downloader. With `start_urls` instead, page 1 came back twice and we got 30 items.

Retries happen in the same place. Here's `scrapy fetch` on an httpbin.dev URL that always returns 503:

shell

```shell
scrapy fetch https://httpbin.dev/status/503
...
2026-11-12 19:43:58 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://httpbin.dev/status/503> (failed 1 times): 503 Service Unavailable
2026-11-12 19:43:58 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://httpbin.dev/status/503> (failed 2 times): 503 Service Unavailable
2026-11-12 19:43:59 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://httpbin.dev/status/503> (failed 3 times): 503 Service Unavailable
```



And here's `requests` on the same URL:

python

```python
response = requests.get("https://httpbin.dev/status/503", timeout=30)
print(response.status_code)
# 503
response.raise_for_status()
# HTTPError: 503 Server Error: Service Unavailable for url: https://httpbin.dev/status/503
```



One attempt, and that's it.

For output, we only changed the file extension:

shell

```shell
scrapy runspider products_spider.py -O products.csv
...
2026-11-12 19:44:15 [scrapy.extensions.feedexport] INFO: Stored csv feed (25 items) in: products.csv
```



```
title,price
Box of Chocolate Candy,24.99
Dark Red Energy Potion,4.99
Teal Energy Potion,4.99
```



By default, Scrapy doesn't include many settings and relies on the built-in defaults, which aren't always optimal.

The selectors, the pagination link and the `start()` request were still ours to write.

But what about the selectors inside those callbacks?



## How do Scrapy selectors differ from Beautiful Soup?

If you've ever done some web development, you're probably familiar with CSS selectors for applying styles to HTML websites - we can use the same tool for parsing HTML data!

For Python, to parse HTML the most popular choices are beautifulsoup4 or parsel. Both take CSS, but they don't run it the same way.

To parse HTML in Python we have several options. However, many of them instead of executing natural CSS selectors convert them to XPath by using [cssselect](https://github.com/scrapy/cssselect) and run the XPath selectors through [lxml](https://lxml.de/) XPath client.

An example of such client would be parsel. Here's what it turns our title selectors into:

python

```python
from parsel.csstranslator import HTMLTranslator

HTMLTranslator().css_to_xpath("h3 a")
# 'descendant-or-self::h3/descendant::a'
HTMLTranslator().css_to_xpath("h3 a::text")
# 'descendant-or-self::h3/descendant::a/text()'
HTMLTranslator().css_to_xpath("h3 a::attr(href)")
# 'descendant-or-self::h3/descendant::a/@href'
```



That `/text()` at the end is all `::text` does.

The availability of pseudo-classes and pseudo-elements varies depending on the client. Here's what we got from Beautiful Soup:

python

```python
card.select("h3 a::text")
# NotImplementedError: Pseudo-element found at position 4
```



You are probably familiar with CSS selectors from the style sheet documents (`.css`) however, XPath goes beyond that and implements full document navigation in its own unique syntax. For HTML parsing, Xpath has some advantages over CSS selectors:

- Xpath can traverse HTML trees in every direction and is location-aware.
- Xpath can transform results before returning them.
- Xpath is easily extendable with additional functionality.

Scrapy has it through `.xpath()`, Beautiful Soup has no XPath at all:

python

```python
card.xpath(".//h3/a/text()").get()
# 'Box of Chocolate Candy'
```



So when do we pick which?



Scrapfly

#### Extract structured data automatically?

Scrapfly's Extraction API uses AI to turn any webpage into structured data — no selectors needed.

[Try Free →](https://scrapfly.io/register)

## When do we pick Beautiful Soup, Scrapy, or both?

When we web scrape, we often have very well-defined list of URLs like "scrape these product web pages on this e-commerce shop".

On the other hand, when we're crawling we have a much looser set of rules like "*find* all product web pages and scrape them on *any* of these websites".

The key difference is that web crawlers are intelligent explorers while web scrapers are focused workers.

### Requests + Beautiful Soup for a direct script

Beautifulsoup is an HTML parsing library so while it's used for web scraping it's not a full web scraping suite/framework like [scrapy](https://www.scrapy.org/).

Beautifulsoup HTML parser needs to be paired with HTTP client library like the [requests](https://pypi.org/project/requests/) library (or other alternatives like [httpx](https://www.python-httpx.org/)) to retrieve HTML pages.

For a focused worker that's all we need.

### Scrapy for a crawl we have to keep

Web crawling is simply more resource-intensive and harder to develop as we need to consider a whole series of new problems related to this exploration component.

We've covered hands-on web scraping with python however, when scaling to hundreds of thousands of requests reinventing the wheel can be a suboptimal and difficult experience.

For big web scraping projects, it might be worth taking a look into web scraping frameworks like [Scrapy](https://www.scrapy.org/) which provides many helper functions and features for various topics we've covered today!

Size isn't everything though. If the crawl has to run again, we use Scrapy.

[Web Scraping With Scrapy: The Complete Guide in 2026Build and run a Scrapy 2.18 project with async start(), pagination, selector tests, item validation, pipelines, and JSON Lines export.](https://scrapfly.io/blog/posts/web-scraping-with-scrapy)

### Scrapy and Beautiful Soup in one spider

So can BeautifulSoup be used with Scrapy? Yes, though scrapy has its own HTML parsing library called [parsel](https://pypi.org/project/parsel/) which is preferred over beautifulsoup4.

The [Scrapy FAQ](https://docs.scrapy.org/en/latest/faq.html#can-i-use-scrapy-with-beautifulsoup) has the recipe for it:

> As mentioned above, BeautifulSoup can be used for parsing HTML responses in Scrapy callbacks. You just have to feed the response's body into a BeautifulSoup object and extract whatever data you need from it.
> 
> Here's an example spider using BeautifulSoup API, with lxml as the HTML parser:

We pointed their example at one of our products:

python

```python
from bs4 import BeautifulSoup
import scrapy


class ExampleSpider(scrapy.Spider):
    name = "example"
    allowed_domains = ["web-scraping.dev"]
    start_urls = ("https://web-scraping.dev/product/1",)

    def parse(self, response):
        # use lxml to get decent HTML parsing speed
        soup = BeautifulSoup(response.text, "lxml")
        yield {"url": response.url, "title": soup.h3.string}
```



```
DEBUG: Scraped from <200 https://web-scraping.dev/product/1>
{'url': 'https://web-scraping.dev/product/1', 'title': 'Box of Chocolate Candy'}
```



But is one of them faster?



## Is Scrapy faster than Beautiful Soup?

In web scraping, we primarily have two types of performance bottlenecks: IO-bound and CPU-bound.

For example, our IO (Input/Output) tasks would be anything that performs an external connection - be it an HTTP request or storing scraped data to a database. Both of which are major parts of web scraper programs.

We also encounter CPU tasks such as parsing scraped HTMLs, loading JSON data, natural language parsing, and so on.

Beautiful Soup only ever does the CPU part, so let's time that first. Here's one parse of our 17,153 character products page, as an average of 200 runs (50 for html5lib):

```
parsel             0.32 ms
bs4 + lxml         2.75 ms
bs4 + html.parser  3.62 ms
bs4 + html5lib     7.52 ms
```



Now the IO part. Here's how long all 5 listing pages took, over three runs each:

```
requests.get()              4.58 - 4.80 s
requests.Session()          1.90 - 1.94 s
Scrapy, 1 request at once   1.95 - 1.99 s
Scrapy, 16 requests at once 1.92 - 2.10 s
```



Milliseconds of parsing next to seconds of waiting.

For more see our article on [web scraping speed](https://scrapfly.io/blog/posts/web-scraping-speed).

But what if the HTML we waited for is the wrong HTML?



## What neither Scrapy nor Beautiful Soup solves alone

Using an HTTP client we may not be able to scrape highly dynamic content like that of javascript-powered web apps or single-page applications (SPAs).

Our `/reviews` page is one of those. Here's the `.review` count from Chrome and from `requests`:

```
https://web-scraping.dev/reviews
chrome    20
requests  0
```



It's likely that these elements are loaded dynamically by javascript. As python doesn't execute javascript it'll not see that content.

To crawl such targets, we need a JavaScript execution context like a headless web browser runner, which can be achieved through browser automation tools (like Playwright, Selenium or Puppeteer).

However, this approach can be difficult and complex and hard to pair with scraping frameworks like Scrapy.

[How to Scrape Dynamic Websites Using Headless Web BrowsersIntroduction to using web automation tools such as Puppeteer, Playwright, Selenium and ScrapFly to render dynamic websites for web scraping](https://scrapfly.io/blog/posts/scraping-using-browsers)

While scrapy is a very powerful and accessible web scraping framework, it doesn't help much with solving the biggest web scraping problem of all - **access blocking**.

Blocks usually come back as a 403 or another 4xx status, but a block page can also arrive with a 200. Here's what that looks like on our `/blocked` page:

python

```python
response = requests.get("https://web-scraping.dev/blocked")
soup = BeautifulSoup(response.text, "lxml")
print(response.status_code, soup.title.string, len(soup.select(".product")))
# 200 web-scraping.dev - You've been blocked 0
```



When the hard part is getting the right HTML, the [Web Scraping API](https://scrapfly.io/products/web-scraping-api) is one way to fetch it.

To wrap up this guide, let's take a look at some frequently asked questions.



## FAQ

Is Beautiful Soup outdated?

No, BeautifulSoup is one of the most popular libraries used in web scraping. It's used to parse HTML documents for data either through Python scripting or the use of CSS selectors.

Version 4.15.0, which we used here, came out in June 2026.







Do I need a full Scrapy project?

No. You can use the `runspider` command. For example, if you have a spider written in a `my_spider.py` file you can run it with:

shell

```shell
scrapy runspider my_spider.py
```









Can Scrapy or Beautiful Soup render JavaScript?

No. As python doesn't execute javascript it'll not see that content. We can use browser automation tools like Selenium though it's hard to make them work well with Scrapy.

Alternatively, a lot of dynamic web page data is actually hidden in the web body, for more see [How to Scrape Hidden Web Data](https://scrapfly.io/blog/posts/how-to-scrape-hidden-web-data#what-is-hidden-web-data).









## Final recommendation

In this article, we've taken an in-depth look into Scrapy and Beautiful Soup and how they differ. We scraped the same products page with both, then timed all 5 listing pages.

For URLs we already have, we'd stay on `requests` or `httpx` with `beautifulsoup4`. For a crawl that has to run again and follow links, we'd use Scrapy.

If our parsing code is already written with `BeautifulSoup`, we've seen that both work together in one spider.

Finally, we've seen where both stop: JavaScript-rendered pages and block pages that return a 200 need a different fetch layer, whichever parser we pick.



Legal Disclaimer and Precautions

This tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect:

- Do not scrape at rates that could damage the website.
- Do not scrape data that's not available publicly.
- Do not store PII of EU citizens protected by GDPR.
- Do not repurpose *entire* public datasets which can be illegal in some countries.

Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer.

 

   [  Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents















 

  Table of Contents

- [Key Takeaways](#key-takeaways)
- [Scrapy vs Beautiful Soup at a glance](#scrapy-vs-beautiful-soup-at-a-glance)
- [Why Scrapy and Beautiful Soup are not direct equivalents](#why-scrapy-and-beautiful-soup-are-not-direct-equivalents)
- [Scrapy vs Beautiful Soup: same page, same output](#scrapy-vs-beautiful-soup-same-page-same-output)
- [Requests + Beautiful Soup](#requests-beautiful-soup)
- [Scrapy in one file](#scrapy-in-one-file)
- [What changes when Beautiful Soup becomes a Scrapy crawl?](#what-changes-when-beautiful-soup-becomes-a-scrapy-crawl)
- [How do Scrapy selectors differ from Beautiful Soup?](#how-do-scrapy-selectors-differ-from-beautiful-soup)
- [When do we pick Beautiful Soup, Scrapy, or both?](#when-do-we-pick-beautiful-soup-scrapy-or-both)
- [Requests + Beautiful Soup for a direct script](#requests-beautiful-soup-for-a-direct-script)
- [Scrapy for a crawl we have to keep](#scrapy-for-a-crawl-we-have-to-keep)
- [Scrapy and Beautiful Soup in one spider](#scrapy-and-beautiful-soup-in-one-spider)
- [Is Scrapy faster than Beautiful Soup?](#is-scrapy-faster-than-beautiful-soup)
- [What neither Scrapy nor Beautiful Soup solves alone](#what-neither-scrapy-nor-beautiful-soup-solves-alone)
- [FAQ](#faq)
- [Final recommendation](#final-recommendation)
 
    Join the Newsletter  Get monthly web scraping insights 

 

  



Scale Your Web Scraping

Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits.

  No credit card required  1,000 free API credits  Anti-bot bypass included 

 [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) 

 Not ready? [Get our newsletter](#newsletter-modal) instead. 

 

 

## Related Articles

 [     

 python data-parsing 

### How to Parse Web Data with Python and Beautifulsoup

Beautifulsoup is one the most popular libraries in web scraping. In this tutorial, we'll take a hand-on overview of how ...

 

 ](https://scrapfly.io/blog/posts/web-scraping-with-python-beautifulsoup) [     

 python nodejs 

### Concurrency vs Parallelism

Learn the key differences between Concurrency and Parallelism and how to leverage them in Python and JavaScript to optim...

 

 ](https://scrapfly.io/blog/posts/concurrency-vs-parallelism) [     

 python 

### Everything to Know to Start Web Scraping in Python Today

Complete introduction to web scraping using Python: http, parsing, AI, scaling and deployment.

 

 ](https://scrapfly.io/blog/posts/everything-to-know-about-web-scraping-python) 

  

## Related Questions

- [ Q What's the difference between Web Scraping and Crawling? ](https://scrapfly.io/blog/answers/whats-the-difference-between-scraping-and-crawling)
- [ Q What are scrapy Item and ItemLoader objects and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-items-and-itemloaders)
- [ Q How to Rotate Proxies in Scrapy ](https://scrapfly.io/blog/answers/scrapy-spiders-proxy-rotation)
- [ Q How to find all links using BeautifulSoup and Python? ](https://scrapfly.io/blog/answers/how-to-find-all-links-using-beautifulsoup)
 
  



   



 Extract structured data with AI, **1,000 free credits** [Start Free](https://scrapfly.io/register)