[Blog](https://scrapfly.io/blog) / [ai](https://scrapfly.io/blog/tag/ai) / [5 Best Scrapy Alternatives for Web Scraping in 2026](https://scrapfly.io/blog/posts/best-scrapy-alternatives-for-web-scraping) # 5 Best Scrapy Alternatives for Web Scraping in 2026 by [Mayada Shaaban](https://scrapfly.io/blog/author/mayada-shaaban-90143e67) Nov 09, 2026 22 min read [\#ai](https://scrapfly.io/blog/tag/ai) [\#headless-browser](https://scrapfly.io/blog/tag/headless-browser) [\#http](https://scrapfly.io/blog/tag/http) [\#python](https://scrapfly.io/blog/tag/python) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-alternatives-for-web-scraping "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-alternatives-for-web-scraping&text=5%20Best%20Scrapy%20Alternatives%20for%20Web%20Scraping%20in%202026 "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-alternatives-for-web-scraping "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-alternatives-for-web-scraping) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-alternatives-for-web-scraping) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-alternatives-for-web-scraping) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-alternatives-for-web-scraping) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-alternatives-for-web-scraping) **Web Scraping API**Scrape any website with anti-bot bypass, proxy rotation, and JS rendering. [ Learn More ](https://scrapfly.io/products/web-scraping-api) [ Docs ](https://scrapfly.io/docs/scrape-api/getting-started) [Scrapy](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) is a great web crawling framework, however it's an HTTP based web scraper which means it doesn't run JavaScript by itself. So when scraping dynamic pages like `web-scraping.dev/reviews` our spider gets a different page than the one we see in the browser - instead of review dates there are only `{{review['date']}}` placeholders, as the reviews are loaded by JavaScript after the page loads. This is usually where web scraper developers start looking for something else to use, even though the rest of Scrapy like crawl scheduling, retries and data exporting is still working just fine. So in this article, we'll take a look at 5 Scrapy alternatives and which piece of Scrapy each of them replaces, with a code example for each one. ## Key Takeaways In this article, we took a glance at five Scrapy alternatives and which part of Scrapy each of them can take over: - Crawlee - the closest thing to Scrapy in the JavaScript world, though running it is still on us. - Playwright - when a page really needs a web browser, though it eats a lot more CPU and memory than an HTTP request. - Beautiful Soup and HTTPX - just fetch the page and parse it, for when a whole Scrapy project is too much (no scheduler or pipelines though). - Crawl4AI - converts web pages to Markdown for LLMs, so there's no HTML to parse anymore. - Scrapling - feels a lot like Scrapy itself, it's just a much younger project. Scrapy, while it can't run JavaScript, is still a great fit for static pages, so there's no need to replace all of it. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## Which Scrapy Alternative Is Best in 2026? Let's take a quick look at how they compare: | Alternative | Best for | Interface | JavaScript | Crawl queue | Primary output | |---|---|---|---|---|---| | Crawlee | JS/TS crawlers | JS/TS, Python | HTTP + browser | Built in | Dataset records | | [Playwright](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python) | Browser workflows | Python, JS, Java, .NET | Native | Bring your own | DOM, screenshots | | [Beautiful Soup](https://scrapfly.io/blog/posts/web-scraping-with-python-beautifulsoup) + [HTTPX](https://scrapfly.io/blog/posts/web-scraping-with-python-httpx) | Small static jobs | Python | None | Bring your own | Python fields | | [Crawl4AI](https://scrapfly.io/blog/posts/crawl4AI-explained) | LLM/RAG ingestion | Python | Browser | Built in | Clean Markdown | | Scrapling | Python spider migration | Python | Optional | Built in | Item records | > Note that we don't have to replace Scrapy entirely! If its scheduler, middlewares, pipelines and feed exports are working fine we can keep them and only replace the part that's giving us trouble. ## Project Setup In this article we'll be using Python for most of the examples and a few popular community packages: - playwright - browser automation library which will let us scrape pages that need JavaScript to load. - httpx - HTTP client library which we'll use to request static pages. - beautifulsoup4 - HTML parsing library that we'll be pairing with httpx. - crawl4ai - crawler which turns web pages into Markdown that LLMs can read. It runs on top of Playwright, so it needs a browser as well. - scrapling - a newer Scrapy-like framework with its own fetchers and spiders. These packages can be installed via `pip` command along with the extra setup commands which download a browser for Playwright, Crawl4AI and Scrapling: bash ```bash pip install playwright httpx beautifulsoup4 crawl4ai "scrapling[fetchers]" playwright install chromium crawl4ai-setup scrapling install ``` Crawlee on the other hand is a JavaScript library, so it's installed through `npm` instead: bash ```bash npm install crawlee ``` As for the target, all of the examples scrape pages from [web-scraping.dev](https://web-scraping.dev/) which is a public website made for web scraping practice - so feel free to run them as they are! ## 1. Crawlee: Best for JavaScript and TypeScript Crawlers Scrapy only comes in Python, which can be a problem when most of our team's code is JavaScript or TypeScript. So can we crawl the Scrapy way without leaving JavaScript? This is where Crawlee comes in handy! It packs crawler classes, a request queue, browser integrations and Dataset storage into one package, which makes it the closest thing to Scrapy we can get in JavaScript. Note that Crawlee comes with a few crawler classes for fetching pages. `CheerioCrawler` works over plain HTTP just like a Scrapy spider, while the Playwright-based crawler is there for pages that need a real browser. Interestingly, Crawlee has a Python edition as well, though the two editions don't have quite the same features. Let's take a look at a small crawler which counts the products on web-scraping.dev's `/products` page. Save it as `crawler.mjs` and run `node crawler.mjs`: javascript ```javascript import { CheerioCrawler, Dataset } from 'crawlee'; const crawler = new CheerioCrawler({ maxRequestsPerCrawl: 1, async requestHandler({ request, $, log }) { const title = $('title').text().trim(); const products = $('div.product').length; log.info(`Found ${products} products on ${request.loadedUrl}`); await Dataset.pushData({ url: request.loadedUrl, title, products, }); }, }); await crawler.run(['https://web-scraping.dev/products']); const { items } = await crawler.getData(); if (items.length !== 1 || items[0].products !== 5) { throw new Error(`Assertion failed: got ${JSON.stringify(items)}`); } console.log(JSON.stringify(items[0], null, 2)); ``` In the example above, our `CheerioCrawler` is limited to a single page and stores what it finds in the default Dataset. Let's break it down: Since `maxRequestsPerCrawl` is set to `1` our crawler stops right after the first page. Our `requestHandler` is where the parsing happens, just like in a Scrapy parse callback. To get the page title we're using `$('title').text()` and to count the products we're using `$('div.product').length`, which `Dataset.pushData()` saves as a Dataset row instead of a Scrapy feed export. Here, `crawler.run()` is replicating Scrapy's scheduler by adding our start URL to Crawlee's request queue, and `crawler.getData()` reads the saved Dataset row back once the crawl is done. Here's what our run with Crawlee 3.18.1 printed: json ```json { "url": "https://web-scraping.dev/products", "title": "web-scraping.dev product page 1", "products": 5 } ``` We can see that our crawler counted all 5 products and saved the result as a single Dataset row! So, with Crawlee we get most of Scrapy's crawling features in JavaScript, though running it is still up to us. In turn, deployment, storage, network quality and monitoring all require our own team's attention. That being said, its browser crawlers are just as resource intensive as Playwright or Puppeteer. Its anti-bot features also need some testing against our own target before we rely on them. [Top Web Crawler Tools in 2026Web crawling has evolved dramatically in 2026. What used to require complex infrastructure and constant maintenance can now be accomplished with a few clicks or lines of code. But with so many options available, choosing the right tool can make or break your data collection project.](https://scrapfly.io/blog/posts/top-web-crawler-tools) ## 2. Playwright: Best for JavaScript-Heavy Browser Workflows Scrapy can only see the HTML that the server sends back, so when the data is loaded by JavaScript our spider returns empty items. This is a pretty common question in Scrapy's own community - here's an [r/scrapy thread](https://www.reddit.com/r/scrapy/comments/1k7h0cf/alternatives_of_scrapy_shell_for_scraping/) asking about exactly that. Using a browser automation library like Playwright we can load these pages in a real browser (Chromium, Firefox or WebKit) and even click, scroll or type on them when the data only shows up after some interaction. Playwright only takes over fetching the pages though, so Scrapy's crawl scheduler and item pipelines have no replacement here. As we've seen in the intro, the reviews on web-scraping.dev's `/reviews` page are loaded by JavaScript, so let's see if Playwright can get them: python ```python import asyncio from playwright.async_api import async_playwright async def main(): async with async_playwright() as p: browser = await p.chromium.launch() page = await browser.new_page() await page.goto("https://web-scraping.dev/reviews") await page.wait_for_selector("div.review") reviews = await page.locator("div.review").all() print(f"Review nodes found: {len(reviews)}") first_text = await page.locator("p[data-testid='review-text']").first.text_content() print(f"First review: {first_text}") assert len(reviews) == 20 await browser.close() asyncio.run(main()) ``` Above, we start a Chromium browser and load the reviews page: - `wait_for_selector()` waits for the first `div.review` element, so we don't count the reviews before the page's JavaScript has loaded them. - `locator("div.review").all()` finds all of the review elements. - `.first.text_content()` reads the text of the first review. If we run it with Playwright 1.62.0 we'll get all 20 reviews: text ```text Review nodes found: 20 First review: Unique flavor and great energy boost. It's the perfect gamer's drink! ``` By using Playwright we don't need to worry about JavaScript anymore, however a browser takes a lot more CPU and memory than an HTTP request (and it won't get around anti-bot protection by itself). So if only some of the pages need it, we can keep Scrapy and send just those through the [Scrapy-Playwright integration](https://scrapfly.io/blog/posts/web-scraping-dynamic-websites-with-scrapy-playwright) instead! ## 3. Beautiful Soup + HTTPX: Best for Simple Python Scrapers A whole crawling framework isn't always needed though. Take a scraper for a handful of pages without JavaScript for example - a Scrapy project with spiders, settings and pipelines is often more work than the scraper itself! Using an HTTP client like HTTPX together with an HTML parser like Beautiful Soup we can scrape these pages in just a few lines of Python. HTTPX takes care of sending the requests, while Beautiful Soup parses the HTML that comes back - it can't fetch any pages on its own. Let's take a look at how we can scrape the product names and prices from web-scraping.dev's `/products` page with these two: python ```python import httpx from bs4 import BeautifulSoup response = httpx.get( "https://web-scraping.dev/products", timeout=10.0, follow_redirects=True, ) response.raise_for_status() soup = BeautifulSoup(response.text, "html.parser") products = soup.select("div.product") for product in products: name = product.select_one("h3").get_text(strip=True) price = product.select_one(".price").get_text(strip=True) print({"name": name, "price": price}) print(f"\nTotal products: {len(products)}") ``` Here, HTTPX fetches the products page and Beautiful Soup's `select()` finds each `div.product`, while `select_one()` reads its name and price with CSS selectors. We also gave the request a `timeout` so it can't hang forever and turned on `follow_redirects` in case the site adds a redirect later. And since we don't want to parse an error page, `raise_for_status()` stops the script if the request fails. If we run this code (we used HTTPX 0.28.1 and Beautiful Soup 4.15.0) we can see all 5 products: text ```text {'name': 'Box of Chocolate Candy', 'price': '24.99'} {'name': 'Dark Red Energy Potion', 'price': '4.99'} {'name': 'Teal Energy Potion', 'price': '4.99'} {'name': 'Red Energy Potion', 'price': '4.99'} {'name': 'Blue Energy Potion', 'price': '4.99'} Total products: 5 ``` By using HTTPX and Beautiful Soup we don't need to set up a whole Scrapy project anymore, however there's no JavaScript support, scheduler or pipelines here (and adding crawl queues, retries and storage ourselves slowly turns it back into a framework). So if the job grows past a few pages, we can always go back to Scrapy instead! Our [Scrapy vs Beautiful Soup comparison](https://scrapfly.io/blog/answers/scrapy-vs-beautifulsoup) covers that choice in more detail. ## 4. Crawl4AI: Best for LLM and RAG Pipelines When scraping data for LLMs or RAG pipelines (like a chatbot that answers questions about a website) we usually don't need the structured items that Scrapy produces. All we really need is the page's text - and raw HTML is full of tags, scripts and styles that only waste the LLM's context. Let's take a look at how we can turn web-scraping.dev's `/products` page into Markdown: python ```python import asyncio from crawl4ai import AsyncWebCrawler async def main(): async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://web-scraping.dev/products") assert result.success markdown = result.markdown.raw_markdown print(f"Markdown length: {len(markdown)} characters") print(markdown[:300]) asyncio.run(main()) ``` Above, we load the products page with `AsyncWebCrawler`: - `arun()` loads the page in a real browser and converts it to Markdown for us. - `assert result.success` makes sure the crawl worked before we read anything. - `raw_markdown` gives us the actual text, as `result.markdown` itself is a wrapper object rather than a string. If we run it with Crawl4AI 0.9.2 we'll get 3,986 characters of Markdown, starting with the site's navigation: text ```text Markdown length: 3986 characters [web-scraping.dev](https://web-scraping.dev/) * [ Docs ](https://web-scraping.dev/products#) * [API](https://web-scraping.dev/docs) * [Graphql](https://web-scraping.dev/api/graphql) * [Products](https://web-scraping.dev/products) * [Reviews](https://web-scraping.dev/reviews) * [Testi ``` Note that Crawl4AI also comes with chunking and structured extraction, though any item schemas or pipelines would have to move to its own extraction settings. So when we need validated item data rather than text, Scrapy is still the better fit. Scrapfly #### Need a cloud browser for scraping? Run headless browsers at scale with Scrapfly Cloud Browser — no infrastructure to manage. [Try Free →](https://scrapfly.io/register)## 5. Scrapling: Best for a Scrapy-Like Python Framework Moving a big Scrapy project to a completely different tool usually means rewriting every spider from scratch - even though most of the time the spiders themselves work just fine. So if we want something newer without changing how our code is structured, we need an alternative that works the same way Scrapy does. Using a framework like [Scrapling](https://scrapling.readthedocs.io/en/latest/) we can keep writing spiders with `start_urls` and an async `parse()` callback, just like in Scrapy. It also comes with its own HTTP and browser fetchers and exporters for JSON, CSV or XML (plus CSS and XPath selectors, concurrency limits and pause and resume), so most of a Scrapy spider's structure carries over as is. Let's take a look at how we can count the products on web-scraping.dev's `/products` page with a Scrapling spider: python ```python from scrapling.spiders import Spider, Response class ProductSpider(Spider): name = "product_spider" start_urls = ["https://web-scraping.dev/products"] async def parse(self, response: Response): title = response.css("title::text").get() products = response.css("div.product") yield {"title": title, "products": len(products)} result = ProductSpider().start() assert len(result.items) == 1, f"expected 1 item, got {len(result.items)}" assert result.items[0]["products"] == 5 print(result.items[0]) print(f"Requests made: {result.stats.requests_count}") ``` Here, we define a `ProductSpider` which looks almost the same as a Scrapy spider would - the `parse()` callback selects the page title and the product elements with CSS selectors and yields one item. The `start()` method doesn't return a plain list though, it returns a `CrawlResult` object, so we read the scraped items from `result.items`. If we run this code (we used Scrapling 0.4.14) we can see the page was scraped in a single request with all 5 products: text ```text {'title': 'web-scraping.dev product page 1', 'products': 5} Requests made: 1 ``` For a quick one-off page outside of a spider there's also `Fetcher.get()`, or `StealthyFetcher.fetch()` when the page needs a browser. By using Scrapling we don't need to rewrite our spiders from scratch anymore, however it's a much younger project than Scrapy. So, we can move one spider over first and test features like adaptive selectors or anti-bot bypass against our own targets. If it holds up, we can migrate the rest of them too! ## Scrapfly Crawler API for Managed Crawl Operations All five of these alternatives still run on our own machines though, so the infrastructure around the crawl is still on us. Running a Scrapy crawler at scale means also running everything around it - request queues, workers, retries and somewhere to store all of the crawled pages. Often, the crawling logic itself is pretty simple and it's this infrastructure that takes up most of our development time. Scrapfly publishes this blog. Using our [Crawler API](https://scrapfly.io/products/crawler-api), we can hand off the crawl infrastructure, as the whole crawl runs on Scrapfly's side instead of our own servers. It offers several features that replace what we'd usually have to set up in Scrapy ourselves: - crawl limits like `page_limit` and `max_depth` will allow us to control how far the crawler goes. - output formats will allow us to get the crawled pages back as HTML, Markdown, JSON or even WARC archives. - JavaScript rendering through `rendering_delay` and Unblocker (previously ASP) through `unblocker=True` for dynamic and protected pages. Unblocker defaults to off; `rendering_delay=0` disables browser rendering. To try it out, we'll be using the Python SDK (`scrapfly-sdk`) which can be installed through pip: bash ```bash pip install scrapfly-sdk ``` Then, let's take a look at a small crawl of web-scraping.dev which is based on the [Crawler API documentation](https://scrapfly.io/docs/crawler-api/getting-started) examples (we didn't run this one here as every crawl is billable): python ```python from scrapfly import ScrapflyClient, CrawlerConfig, Crawl client = ScrapflyClient(key="YOUR SCRAPFLY KEY") crawl = Crawl(client, CrawlerConfig( url="https://web-scraping.dev/products", page_limit=5, content_formats=["markdown"], )).crawl().wait() content = crawl.read("https://web-scraping.dev/products", format="markdown") if content: print(content.content[:200]) ``` Above, we start a crawl from the `/products` page and use the SDK's `Crawl` object for the rest of it: - `page_limit` stops the crawl after 5 pages. - `wait()` blocks until the crawl is finished. - `read()` retrieves the `/products` page back as Markdown. Note that Scrapy's middlewares and item pipelines don't come along, so that logic would have to move to webhooks or our own code that runs after the crawl. For more on when to use the Crawler API over Scrapfly's Web Scraping API see our [Crawler API vs Web Scraping API comparison](https://scrapfly.io/blog/posts/scraper-api-vs-crawler-api). ### Web Scraping API Scrape any website with our powerful API. Anti-bot bypass, JavaScript rendering, and rotating proxies built-in. [Try Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started) ## Should You Replace Scrapy or Extend It? Let's take a look at what we can keep from Scrapy for each of the common problems: | Current bottleneck | Smallest change | Keep from Scrapy | Replacement path | |---|---|---|---| | A few JavaScript pages | Add browser requests | Scheduler, pipelines | Playwright | | Entire workflow needs browser actions | Move interaction flow | Downstream schema | Playwright | | Small static URL set | Remove the framework | Field schema | Beautiful Soup + HTTPX | | JavaScript/TypeScript team | Change crawler runtime | Output contract | Crawlee | | LLM-ready Markdown | Change primary output | URL scope | Crawl4AI | | Python team wants a newer spider model | Port one spider | Selectors, item shape | Scrapling | | Crawler operations burden | Move crawl operations | Downstream processing | Managed crawler API | A pretty common mistake when a selector stops matching is adding a browser right away. Often the data isn't missing at all though, and we're simply looking at a block page. Let's take a look at how that plays out on web-scraping.dev's `/blocked` page: python ```python import httpx from bs4 import BeautifulSoup for url in ["https://web-scraping.dev/products", "https://web-scraping.dev/blocked"]: response = httpx.get(url, timeout=10.0, follow_redirects=True) soup = BeautifulSoup(response.text, "html.parser") print(url) print(f" status={response.status_code} final_url={response.url}") print(f" products={len(soup.select('div.product'))} title={soup.title.get_text(strip=True) if soup.title else None!r}") ``` Here, we're requesting the regular `/products` page and the `/blocked` page with the same httpx call. Then, for each response we print the final URL, count the `div.product` elements and grab the page title. This will allow us to see what each page actually returned - and if we run it, we'll get a `200` status for both: text ```text https://web-scraping.dev/products status=200 final_url=https://web-scraping.dev/products products=5 title='web-scraping.dev product page 1' https://web-scraping.dev/blocked status=200 final_url=https://web-scraping.dev/blocked products=0 title="web-scraping.dev - You've been blocked" ``` In the output above, we can see that both responses look fine by their status code. However, our selector found 0 products on `/blocked` and only the page title gives the block away. Meaning, adding a browser here would just be chasing the wrong problem! By adding this same check to a Scrapy downloader middleware we can catch the block page before our spider ever sees it, as the middleware gets every response first. Though, most importantly it allows us to fix the problem without switching frameworks (see Scrapy's [extensions and middleware system](https://scrapfly.io/blog/posts/best-scrapy-extensions-middlewares)). ## How Do You Migrate a Scrapy Project Without Rewriting Everything? First, we could rewrite the whole Scrapy project in a new framework. Unfortunately, as the project grows, this approach becomes very time-consuming, difficult and has to be done for every spider at once. Alternatively, we can replace Scrapy's parts like spiders, middlewares and pipelines one by one, as most of them can be replaced on their own: | Scrapy component | Preserve first | Replacement equivalent | |---|---|---| | Spider traversal | URL scope and stop rules | Crawlee/Scrapling/API scope | | Downloader/middleware | Headers, sessions, retries | HTTPX/browser/API config | | Selectors | Field names and fixtures | BS4/locators/extraction/selectors | | Item pipelines | Validation and schema | Dataset/Python/downstream validation | | Feed exports | File and record contract | Dataset/JSONL/Markdown/WARC | | Stats and logs | Failure counters and alerts | Runtime/API/external monitoring | To make sure the new part works, we can run the old and the new version side by side. For example, if we've moved a spider over to Scrapling we'd run both of them on the same page and compare the fields, item count, retries and run time. If it's pretty much the same we can move on to the next part, however if it isn't we still have the old Scrapy spider to fall back to! [How to Crawl the Web with PythonIntroduction to web crawling with Python. What is web crawling? How it differs from web scraping? And a deep dive into code, building our own crawler and an example project crawling Shopify-powered websites.](https://scrapfly.io/blog/posts/crawling-with-python) Note: none of this is a benchmark, as the only thing we've timed is HTTPX vs Playwright in the next section. Meaning, we didn't test blocking at all. ## Where Do Scrapy Alternatives Still Break? Of course, none of these alternatives are perfect either. Take Playwright for example - to render a page it runs a whole Chromium browser in the background, and that browser downloads a lot more than just the HTML. Let's load the same `/products` page with both HTTPX and Playwright and see: python ```python import httpx from time import time from playwright.sync_api import sync_playwright url = "https://web-scraping.dev/products" # plain HTTP request - only the HTML is downloaded _start = time() response = httpx.get(url) print(f"httpx: {time() - _start:.2f}s, {len(response.text)} bytes") # browser request - the HTML and everything the page loads with it _start = time() with sync_playwright() as pw: browser = pw.chromium.launch() page = browser.new_page() requests = [] # record every request the browser sends: page.on("request", lambda request: requests.append(request.url)) page.goto(url) html = page.content() browser.close() print(f"playwright: {time() - _start:.2f}s, {len(html)} bytes, {len(requests)} requests") ``` Here, we're loading the same page twice and timing each one. For the browser we also count how many requests Chromium sends with the `page.on("request")` callback. If we run it with HTTPX 0.28.1 and Playwright 1.62.0 we'll get: text ```text httpx: 0.89s, 17153 bytes playwright: 2.22s, 17077 bytes, 13 requests ``` The HTML is pretty much the same, however the browser also loaded the page's stylesheets, scripts and images and took over twice as long - so for thousands of static pages Playwright would just slow our crawl down! The other alternatives have their own catches too, which we've mentioned at the end of each section above - like our Beautiful Soup scraper slowly turning back into a framework once we add retries and queues to it. Scrapling's adaptive selectors are the only feature we didn't try at all here, so we can't say much about how well they work yet! ## FAQ Is Beautiful Soup an alternative to Scrapy?Not by itself, as Beautiful Soup can't download pages. If we give it a URL it just parses the URL string as if it was HTML: python ```python from bs4 import BeautifulSoup soup = BeautifulSoup("https://web-scraping.dev/products", "html.parser") print(soup.select(".product")) ``` text ```text MarkupResemblesLocatorWarning: The input passed in on this line looks more like a URL than HTML or XML. [] ``` That's why the Beautiful Soup example above uses HTTPX to get the page first. Can Playwright take over for Scrapy?Yes, as far as loading pages goes. Our Playwright example got all 20 reviews from `/reviews`, while Scrapy only sees template placeholders on the same page. Though, that example loads just one page - for crawling we'd still need Scrapy with the [Scrapy-Playwright integration](https://scrapfly.io/blog/posts/web-scraping-dynamic-websites-with-scrapy-playwright). Is Crawlee better than Scrapy?Only if we're working in JavaScript. Our `CheerioCrawler` example does what a small Scrapy spider does, just in Node.js. What is the best Scrapy alternative for LLM and RAG pipelines?We'd go with Crawl4AI. Inside the earlier example's `main()` function, add these two lines after `print(markdown[:300])`, at the same indentation level, to print the HTML and Markdown lengths: python ```python print("html:", len(result.html)) print("markdown:", len(result.markdown.raw_markdown)) ``` text ```text html: 17076 markdown: 3986 ``` That's 13,090 fewer characters in the Markdown, which makes it about 4 times smaller - and far fewer tokens to send to the LLM! Is Scrapy still worth using in 2026?Yes. If our spiders are scraping static pages without problems there's no reason to move away from Scrapy. Is it legal to scrape websites with these tools?Generally yes, as long as we scrape public data at a respectful rate and stay away from personal or copyrighted data. For more see our [Is Web Scraping Legal?](https://scrapfly.io/is-web-scraping-legal) article. ## Summary In this overview article, we took a glance at the most popular Scrapy alternatives and which parts of Scrapy they can take over. We took a look at Crawlee - the closest thing to Scrapy in the JavaScript world, Playwright for scraping pages that need a real web browser and the classic Beautiful Soup and HTTPX combo for when a whole crawling framework is just too much. Then we covered two newer Python tools - Crawl4AI, which converts web pages to Markdown for LLMs, and Scrapling, which feels a lot like Scrapy itself so moving spiders over is pretty easy. Scrapy, while lacking JavaScript support, is still a great crawling framework for most web scraping projects. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents - [Key Takeaways](#key-takeaways) - [Which Scrapy Alternative Is Best in 2026?](#which-scrapy-alternative-is-best-in-2026) - [Project Setup](#project-setup) - [1. Crawlee: Best for JavaScript and TypeScript Crawlers](#1-crawlee-best-for-javascript-and-typescript-crawlers) - [2. Playwright: Best for JavaScript-Heavy Browser Workflows](#2-playwright-best-for-javascript-heavy-browser-workflows) - [3. Beautiful Soup + HTTPX: Best for Simple Python Scrapers](#3-beautiful-soup-httpx-best-for-simple-python-scrapers) - [4. Crawl4AI: Best for LLM and RAG Pipelines](#4-crawl4ai-best-for-llm-and-rag-pipelines) - [5. Scrapling: Best for a Scrapy-Like Python Framework](#5-scrapling-best-for-a-scrapy-like-python-framework) - [Scrapfly Crawler API for Managed Crawl Operations](#scrapfly-crawler-api-for-managed-crawl-operations) - [Should You Replace Scrapy or Extend It?](#should-you-replace-scrapy-or-extend-it) - [How Do You Migrate a Scrapy Project Without Rewriting Everything?](#how-do-you-migrate-a-scrapy-project-without-rewriting-everything) - [Where Do Scrapy Alternatives Still Break?](#where-do-scrapy-alternatives-still-break) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ python ### Everything to Know to Start Web Scraping in Python Today Complete introduction to web scraping using Python: http, parsing, AI, scaling and deployment. ](https://scrapfly.io/blog/posts/everything-to-know-about-web-scraping-python) [ ### Crawl4AI Guide: Web Crawling for LLMs, RAG, and AI Agents Learn how to use Crawl4AI v0.8.x for AI-ready web crawling.Covers installation, LLM extraction with Pydantic, deep crawl... ](https://scrapfly.io/blog/posts/crawl4AI-explained) [ python httpx ### How to Web Scrape with HTTPX and Python Intro to using Python's httpx library for web scraping. Proxy and user agent rotation and common web scraping challenges... ](https://scrapfly.io/blog/posts/web-scraping-with-python-httpx) ## Related Questions - [ Q How to use proxies with Python httpx? ](https://scrapfly.io/blog/answers/how-to-use-proxies-python-httpx) - [ Q Scrapy vs BeautifulSoup: Which Should You Use? ](https://scrapfly.io/blog/answers/scrapy-vs-beautifulsoup) Run headless browsers at scale, **1,000 free credits** [Start Free](https://scrapfly.io/register)