[Blog](https://scrapfly.io/blog) / [headless-browser](https://scrapfly.io/blog/tag/headless-browser) / [9 Scrapy Extensions and Middlewares for 2026](https://scrapfly.io/blog/posts/best-scrapy-extensions-middlewares) # 9 Scrapy Extensions and Middlewares for 2026 by [Mayada Shaaban](https://scrapfly.io/blog/author/mayada-shaaban-90143e67) Sep 10, 2026 17 min read [\#headless-browser](https://scrapfly.io/blog/tag/headless-browser) [\#proxies](https://scrapfly.io/blog/tag/proxies) [\#python](https://scrapfly.io/blog/tag/python) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares&text=9%20Scrapy%20Extensions%20and%20Middlewares%20for%202026 "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares) Scrapy 2.18 changed the startup and spider-middleware contracts. Several familiar package picks now fail before the first useful response. A package that imported cleanly on Scrapy 2.12 can raise a `TypeError` at crawler startup on 2.18. This list covers nine packages by job: rendering, TLS fingerprints, proxy rotation, reusable extraction, a shared crawl queue, and deployment. scrapy-selenium4 and scrapy-splash appear as maintenance choices, not picks for a new build. [Web Scraping With Scrapy: The Complete Guide in 2026Build and run a Scrapy 2.18 project with async start(), pagination, selector tests, item validation, pipelines, and JSON Lines export.](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) ## Key Takeaways - **Install by gap.** Add whatever component fills the missing layer. - **scrapy-playwright leads rendering.** The default pick, tested clean on Scrapy 2.18.0. - **Selenium and Splash stay pinned.** Maintenance-only choices, not new-build defaults. - **scrapy-impersonate matches TLS.** Spoofs fingerprints without JavaScript, not a bypass. - **scrapy-redis and scrapy-poet split jobs.** Shared queue versus reusable extraction. - **Scrapyd runs, scrapyd-client ships.** The server runs jobs, the client deploys versions. - **A PyPI date proves nothing.** scrapy-deltafetch still fails at Scrapy 2.18 startup. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## Install by Missing Layer: Rendering, Fingerprints, Proxies, or Queue Install [scrapy-playwright](https://github.com/scrapy-plugins/scrapy-playwright) for selective browser rendering. Add [scrapy-impersonate](https://github.com/jxlil/scrapy-impersonate) for HTTP fingerprint control, [scrapy-redis](https://github.com/rolando/scrapy-redis) for a shared queue and duplicate filter, and the [Scrapyd](https://scrapyd.readthedocs.io/en/stable/) tools for deployment. These nine components form a stack ordered by workflow, not nine substitutes for the same job. Each row names the reader problem it solves and the boundary the test covered. | \# | Component | Reader job | Primary output | Tested boundary | |---|---|---|---|---| | 1 | scrapy-playwright | Render JavaScript | Rendered response | 2.18.0 + 0.0.48 | | 2 | scrapy-selenium4 | Keep Selenium flow | Selenium response | 2.18.0 + warning | | 3 | scrapy-splash | Keep Splash flow | HTTP or Lua render | 2.12.0 pinned | | 4 | scrapy-impersonate | Match TLS profile | HTTP response | 2.18.0 + 1.8.1 | | 5 | scrapy-rotating-proxies | Rotate raw proxies | Proxy lifecycle | 2.18.0 + 0.6.2 | | 6 | scrapy-poet | Reuse extraction | Injected page object | 2.18.0 + 0.27.2 | | 7 | scrapy-redis | Share crawl state | Redis queue and set | 2.18.0 + 0.9.1 | | 8 | Scrapyd | Run crawl jobs | Job API and logs | 2.18.0 + 1.6.0 | | 9 | scrapyd-client | Ship project builds | Uploaded project version | 2.18.0 + 2.0.3 | Read the table top to bottom by what your crawl is missing today, not by package popularity. A crawl that only needs raw HTTP has no reason to add a browser handler. A single-machine crawl has no reason to add scrapy-redis. Plain Scrapy covers the request and response cycle these packages plug into. ## Add-ons, Middlewares, Download Handlers, Schedulers, and Services Own Different Settings Each integration point owns a different part of the crawl lifecycle. Its class belongs under the setting that package documents, not wherever it happens to import cleanly. Mixing these up is the most common configuration error in this list. | Integration point | Setting or boundary | What it changes | |---|---|---| | Add-on | `ADDONS` | Several settings as one unit | | Extension | `EXTENSIONS` | Signals, stats, and monitoring | | Downloader middleware | `DOWNLOADER_MIDDLEWARES` | Requests and responses | | Spider middleware | `SPIDER_MIDDLEWARES` | Spider input and output | | Download handler | `DOWNLOAD_HANDLERS` | Network transport | | Scheduler | `SCHEDULER` | Request queue and priority | | Service or CLI | Separate process | Running or exposing spiders | scrapy-playwright and scrapy-impersonate both swap in a download handler. They own network transport per scheme rather than touching every request through a middleware hook. scrapy-redis replaces the scheduler and duplicate-filter classes instead, and Scrapyd runs as a separate service with its own process and API. Getting a package's class into the wrong setting is a fast way to have it silently do nothing. ## Project Setup Each section below assumes a working Scrapy project on version 2.18.0. Install the package for whichever section you're testing before running its snippet: bash```bash pip install scrapy==2.18.0 scrapy-playwright==0.0.48 playwright install chromium ``` Swap in the matching package name (`scrapy-impersonate`, `scrapy-redis`, and so on) for the section you're reading, and check its own version pin against the table above. ## 1. scrapy-playwright: Best for Selective JavaScript Rendering scrapy-playwright 0.0.48 is the default choice when selected Scrapy requests need a current [Playwright](https://playwright.dev/) browser. The tested boundary is Scrapy 2.18.0, scrapy-playwright 0.0.48, and Playwright 1.62.0. It ran Chrome for Testing 151.0.7922.34 as the Chromium build. It works as a download handler activated per request through `meta={"playwright": True}`, so plain HTTP requests in the same spider skip the browser entirely. Inspect the initial HTML and the page's background requests before reaching for this handler. Many pages that look JavaScript-rendered expose the same data through a plain XHR call a normal request can read directly. python```python import scrapy from scrapy.crawler import CrawlerProcess SETTINGS = { "DOWNLOAD_HANDLERS": { "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", }, "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", } class ProductsSpider(scrapy.Spider): name = "products_playwright" async def start(self): yield scrapy.Request( "https://web-scraping.dev/products", meta={"playwright": True}, ) def parse(self, response): cards = response.css(".product") for card in cards: yield { "title": card.css("h3 a::text").get(default="").strip(), "price": card.css(".price::text").get(default="").strip(), } if __name__ == "__main__": process = CrawlerProcess(SETTINGS) process.crawl(ProductsSpider) process.start() ``` This spider requests `web-scraping.dev/products` through the Playwright download handler, waits for the rendered response, and yields the title and price scraped from each product card. This run against `web-scraping.dev/products` returned five items through the rendered response and normal Scrapy selectors: json```json [ {"title": "Box of Chocolate Candy", "price": "24.99"}, {"title": "Dark Red Energy Potion", "price": "4.99"}, {"title": "Teal Energy Potion", "price": "4.99"}, {"title": "Red Energy Potion", "price": "4.99"}, {"title": "Blue Energy Potion", "price": "4.99"} ] ``` Browser requests consume more CPU and memory per page than plain HTTP. Running a browser at all does not guarantee anti-bot access on a protected target. For a full walkthrough of waits, browser contexts, and page lifecycle, see [the dedicated scrapy-playwright guide](https://scrapfly.io/blog/posts/web-scraping-dynamic-websites-with-scrapy-playwright). One boundary sits outside the scrapy-playwright handler entirely. Direct Playwright can connect to a [Cloud Browser](https://scrapfly.io/products/cloud-browser-api) session over CDP once operating browser sessions become their own problem, separate from the scraping logic. A Playwright 1.62 script connecting over CDP reached `https://web-scraping.dev/products`, returned HTTP 200, and counted five product cards. Grab the session's existing context with `browser.contexts[0]` rather than opening a fresh one. The CDP script in this article's companion file falls back to `browser.new_context()` only when no context exists, and reuses the existing context otherwise. That's a direct Playwright-to-Cloud Browser connection over CDP, not a scrapy-playwright download-handler configuration. It only applies once browser sessions are the operational concern. ## 2. scrapy-selenium4: Best for Maintaining Selenium-Based Scrapy Crawlers Use `scrapy-selenium4==1.0.0` only when an existing project already depends on [Selenium](https://www.selenium.dev/) WebDriver behavior inside Scrapy. It is a maintenance choice, not a starting point for a new browser integration. This is a different package from the original `scrapy-selenium==0.0.7`, which lives at a different repository and targets Selenium 3. The reproduced matrix here is Scrapy 2.18.0, Selenium 4.47.0, and Chrome and ChromeDriver both on major version 151. Selenium Manager resolved that browser and driver version automatically, rather than pinning them by hand. The package's middleware still uses the two-argument `process_request(self, request, spider)` signature Scrapy is phasing out. Scrapy raises a `ScrapyDeprecationWarning` once, at crawler startup, when it registers that method and finds the `spider` parameter still required. Its repository, a fork at `zhouyangchao/scrapy-selenium`, last committed in September 2024 and carries a single star with issues disabled: a small maintenance surface by any measure. Browser work inside the middleware also runs synchronously, blocking the reactor for each page load. For a new browser integration, use scrapy-playwright instead. This entry is a status and migration decision, not a second Selenium tutorial. See [the existing Scrapy Selenium guide](https://scrapfly.io/blog/posts/web-scraping-dynamic-web-pages-with-scrapy-selenium) for the working setup this maintenance path assumes. ## 3. scrapy-splash: Best for Maintaining a Pinned Splash Stack Keep scrapy-splash only for an existing simple-render or Lua workflow that can stay on a frozen stack. `scrapy-splash==0.11.1` fails to import on Scrapy 2.18: it imports `canonicalize_url` from `scrapy.utils.url`, and current Scrapy no longer exposes that function there (it now lives in `w3lib.url`). The reproduced working baseline is Python 3.12, Scrapy 2.12.0, Twisted 24.11.0, scrapy-splash 0.11.1, and Splash 3.5. Splash itself is a separate HTTP rendering service, not a Python library that runs in-process. Its 3.5 server ships a legacy browser engine that predates current Chromium-based rendering. Under that pinned stack the import succeeds, and a `SplashRequest` against a local Splash service returned a fully rendered page. Route new browser work to scrapy-playwright instead. This entry stays a status check rather than a setup tutorial, since a frozen stack shouldn't get presented as a fresh recommendation. See [the dedicated Scrapy Splash guide](https://scrapfly.io/blog/posts/web-scraping-with-scrapy-splash) for the full pinned setup and Lua examples. ## 4. scrapy-impersonate: Best for Browser-Like TLS Fingerprints Without a Browser scrapy-impersonate 1.8.1 replaces Scrapy's HTTP download handler with curl\_cffi. A request can then present a browser-like TLS and HTTP fingerprint without running JavaScript. The tested boundary is Scrapy 2.18.0 with `impersonate="chrome146"`, a real curl\_cffi profile in this release. It returned HTTP 200 and a non-empty JA3 hash from a public TLS fixture. Leave `USER_AGENT` empty so curl\_cffi picks a value that matches the impersonation profile automatically, and keep the asyncio reactor active for the handler to load. Redirect handling is worth calling out: this handler has no bare `allow_redirects` request argument the way `requests` or `curl_cffi` do. Redirects stay under Scrapy's own `RedirectMiddleware` by default. Only pass `impersonate_args={"allow_redirects": True}` if you deliberately want curl\_cffi to follow redirects itself, which bypasses Scrapy's redirect and offsite middleware. python```python import json import scrapy class ImpersonateSpider(scrapy.Spider): name = "impersonate_test" custom_settings = { "DOWNLOAD_HANDLERS": { "http": "scrapy_impersonate.ImpersonateDownloadHandler", "https": "scrapy_impersonate.ImpersonateDownloadHandler", }, "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", "USER_AGENT": "", } async def start(self): meta = {"impersonate": "chrome146"} yield scrapy.Request("https://tls.peet.ws/api/all", meta=meta) def parse(self, response): data = json.loads(response.text) ja3_hash = data.get("tls", {}).get("ja3_hash") self.logger.info("STATUS=%s JA3_HASH=%s", response.status, ja3_hash) yield {"status": response.status, "ja3_hash": ja3_hash} ``` Two separate runs against the live fixture both returned status 200 with a populated hash, for example `ja3_hash: "5c9265d5eed9cdeb0cbd9c3fd2111388"` on one run. The hash itself varies between runs and targets, so treat it as a live signal to check, not a constant to hard-code. This changes HTTP transport signals only, not JavaScript execution, cookies earned through page interaction, or human behavior patterns. A matching JA3 hash does not identify every browser or clear every anti-bot check on its own. See [TLS fingerprint blocking](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-tls) for how targets use this signal alongside others. ## 5. scrapy-rotating-proxies: Best for a Small Self-Managed Proxy List scrapy-rotating-proxies 0.6.2 can still assign and cool down members of a raw proxy list. It needs a pinned compatibility check and a target-specific ban policy before you rely on it. The maintainer uploaded the release to PyPI on May 25, 2019, and its packaging still declares an unpinned dependency on the old `typing` backport. That backport doesn't block installation on current Python, contrary to what its age might suggest. It installed cleanly alongside Scrapy 2.18.0 on both Python 3.13 and 3.14 in a fresh check. `typing`'s own metadata carries no upper version bound, so pip rebuilds it from source. The real caveat is packaging hygiene, not a hard install block. Under Scrapy 2.18 the middleware loaded correctly and rotated requests across both members of a two-proxy test fixture. That run also logged a `ScrapyDeprecationWarning` about its methods needing an explicit `spider` argument in a future Scrapy release. A 403 or 429 response describes what the target returned, not automatically a dead proxy. Pair this package with a policy that fits your target rather than trusting its defaults. A minimal setup adds `rotating_proxies.middlewares.RotatingProxyMiddleware` at priority 610 and `BanDetectionMiddleware` at 620 under `DOWNLOADER_MIDDLEWARES`, with `ROTATING_PROXY_LIST` pointed at your own proxy addresses. For the full proxy-list lifecycle and ban-detection tuning, see [proxy rotation strategies](https://scrapfly.io/blog/posts/how-to-rotate-proxies-in-web-scraping). This entry does not change TLS or browser fingerprints. It only changes which upstream IP a request leaves from. Scrapfly #### Need a cloud browser for scraping? Run headless browsers at scale with Scrapfly Cloud Browser — no infrastructure to manage. [Try Free →](https://scrapfly.io/register)## 6. scrapy-poet: Best for Reusable Extraction Across Many Sites scrapy-poet 0.27.2 separates extraction into injected page objects, so one spider can reuse traversal logic across sites with different page layouts. Activate it with `ADDONS = {"scrapy_poet.Addon": 300}` on Scrapy 2.10 and newer. On Scrapy 2.18 that add-on injected a `WebPage`-based object into a callback and returned the expected record from a live page. python```python import scrapy from web_poet import WebPage from scrapy.crawler import CrawlerProcess class ProductPage(WebPage): async def to_item(self) -> dict: return { "name": self.css("h3.product-title::text").get("").strip(), "price": self.css("span.product-price::text").get("").strip(), } class PoetSpider(scrapy.Spider): name = "poet_test" custom_settings = {"ADDONS": {"scrapy_poet.Addon": 300}} async def start(self): yield scrapy.Request("https://web-scraping.dev/product/1", callback=self.parse) async def parse(self, response, page: ProductPage): item = await page.to_item() self.logger.info("EXTRACTED_ITEM=%r", item) yield item if __name__ == "__main__": process = CrawlerProcess() process.crawl(PoetSpider) process.start() ``` The run against `web-scraping.dev/product/1` logged `EXTRACTED_ITEM={'name': 'Box of Chocolate Candy', 'price': '$9.99'}`, and the crawl stats confirmed the injector actually constructed `ProductPage` rather than passing through a plain response. The output here is an independently testable extraction object, not faster downloads or a shared crawl queue. Those are separate jobs further down this list. ## 7. scrapy-redis: Best for a Shared Queue and Duplicate Filter scrapy-redis 0.9.1 gives multiple Scrapy workers a [Redis](https://redis.io/)-backed request queue, duplicate filter, and optional item pipeline. On Scrapy 2.18, the priority queue round-tripped a `scrapy.Request` with its URL, priority, and metadata intact. The Redis duplicate filter correctly rejected the same request fingerprint on its second check. python```python import os os.environ.setdefault("REDIS_URL", "redis://localhost:6379/0") REDIS_URL = os.environ["REDIS_URL"] SCHEDULER = "scrapy_redis.scheduler.Scheduler" DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter" SCHEDULER_PERSIST = True SCHEDULER_QUEUE_KEY = "myproject:%(spider)s:requests" SCHEDULER_DUPEFILTER_KEY = "myproject:%(spider)s:dupefilter" ``` Every worker needs to share the same Redis keys and project settings for queue and deduplication behavior to stay coherent between processes. A worker pointed at a different key prefix won't see the others' requests. This package supplies shared queue and fingerprint state, not an advanced crawl-frontier policy, worker deployment, or exactly-once processing guarantees. A worker can still reprocess a request it dequeues but crashes before finishing. [Concurrency vs ParallelismLearn the key differences between Concurrency and Parallelism and how to leverage them in Python and JavaScript to optimize performance in various computational tasks.](https://scrapfly.io/blog/posts/concurrency-vs-parallelism) ## 8. Scrapyd: Best for Running Long-Lived Scrapy Jobs Scrapyd 1.6.0 is the deployment target when Scrapy projects need a daemon and JSON API for uploaded versions, scheduled jobs, logs, and items. Running it beside Scrapy 2.18 and querying `daemonstatus.json` returned a live `{"status": "ok", ...}` response, and it accepted a project upload through its deploy endpoint without issue. Scrapyd is the server component only. It doesn't build or upload a project by itself. That's scrapyd-client's job, covered next. Keep production hardening, process supervision, and network access control out of a quick setup. Scrapyd's own docs cover running it behind a reverse proxy with authentication. None of that belongs on a daemon exposed directly to the internet. ## 9. scrapyd-client: Best for Packaging and Uploading Scrapy Projects scrapyd-client 2.0.3 builds a Scrapy project artifact, uploads it through Scrapyd's `addversion.json` API, and can query or schedule the project it deployed. With Scrapy 2.18 and Scrapyd 1.6.0 running locally, `scrapyd-client deploy` built the project and received a 200 response with `status: ok`. ini```ini # scrapy.cfg [deploy:local] url = ${SCRAPYD_URL} project = demo ``` bash```bash export SCRAPYD_URL=http://localhost:6800 scrapyd-client deploy local -p demo scrapyd-client projects -t local ``` The `scrapy.cfg` block names a deploy target and points it at the running Scrapyd instance. The commands below it export that URL, build and upload the project to `demo`, then list the projects registered on that target. Version 2.0.3 ships a unified `scrapyd-client` command with `deploy`, `targets`, `projects`, and `spiders` subcommands, alongside the older standalone `scrapyd-deploy`script that still works through the same code path. This release drops the legacy `-l` and `-L` list flags on `scrapyd-deploy`. Use `scrapyd-client projects` and `scrapyd-client spiders` instead. Environment-variable interpolation in `scrapy.cfg` works natively, so keep the URL out of the file and export it at deploy time rather than committing it. The output here is a versioned project available to Scrapyd, not a running crawl until something schedules it. ## Which Scrapy Package Recommendations Are Stale on Scrapy 2.18? A recent release date does not prove compatibility with Scrapy 2.18's asynchronous spider-output contract. Two packages that still show up in search results and generated answers fail outright once a project upgrades. | Package | Current finding | Safer route | |---|---|---| | scrapy-deltafetch 2.1.0 | Fails at crawler startup | Redis duplicate filter or native job state | | scrapy-selenium 0.0.7 | Selenium 3 constructor path | scrapy-selenium4 for maintenance | | scrapy-splash 0.11.1 | Import fails on Scrapy 2.18 | Pinned Scrapy 2.12 or Playwright | scrapy-deltafetch's spider middleware implements a plain synchronous `process_spider_output`, and Scrapy 2.18 requires spider middlewares to support asynchronous output. Loading it raises `TypeError: Middleware scrapy_deltafetch.middleware.DeltaFetch doesn't support asynchronous spider output` at crawler startup, before a single request goes out. scrapy-selenium 0.0.7 still instantiates its WebDriver with `executable_path` and `chrome_options` keyword arguments, which Selenium 4 removed. That's the constructor path scrapy-selenium4 exists to fix. This table can't prove any of these three projects sit abandoned, and it makes no claim about maintainer intent. It states the package, the version, the release date where relevant, and the behavior observed. ## Power-up with Scrapfly Every extension in this list still leaves rendering, fingerprinting, and proxy rotation as separate packages to install and monitor. Scrapfly's [Web Scraping API](https://scrapfly.io/products/web-scraping-api) is a single HTTP endpoint for collecting web data at scale, with a **98% success rate** across **130M+ proxies in 190+ countries**. - [Anti-Scraping Protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - automatically defeats Cloudflare, DataDome, PerimeterX, Akamai, and 90+ other bot systems. - [Smart proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) - residential and datacenter pools with country and ASN level geo-targeting. - [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - render SPAs and dynamic pages through real cloud browsers. - [Browser automation scenarios](https://scrapfly.io/docs/scrape-api/javascript-scenario) - scroll, click, fill forms, and wait for elements without managing a browser fleet. - [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - return pages as HTML, JSON, clean text, or LLM ready Markdown. - [Session management](https://scrapfly.io/docs/scrape-api/session) - keep cookies, headers, and IPs consistent across multi step flows. - [Smart caching](https://scrapfly.io/docs/scrape-api/getting-started#api_param_cache) - cache successful responses to cut cost on repeat scraping jobs. - [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Scrapy](https://scrapfly.io/docs/sdk/scrapy), and [no-code integrations](https://scrapfly.io/docs/integration/getting-started) including [Make](https://scrapfly.io/integration/make), [n8n](https://scrapfly.io/integration/n8n), [Zapier](https://scrapfly.io/integration/zapier), [LangChain](https://scrapfly.io/integration/langchain), and [LlamaIndex](https://scrapfly.io/integration/llamaindex). The Scrapy SDK drops into the same `DOWNLOAD_HANDLERS` setting this article's packages use, so switching to Scrapfly doesn't mean rewriting existing spider logic. ### Web Scraping API Scrape any website with our powerful API. Anti-bot bypass, JavaScript rendering, and rotating proxies built-in. [Try Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started) ## FAQ Can scrapy-playwright, scrapy-impersonate, and scrapy-redis run in the same project?Yes, but not by accident. scrapy-redis replaces the scheduler and duplicate filter, while scrapy-playwright and scrapy-impersonate are both download handlers competing for the same `http` and `https` keys. Only one can own a scheme, so route the other per request through its own `meta` key or a separate scheme entry. Does scrapy-impersonate bypass every anti-bot system?No. It changes TLS and HTTP/2 fingerprints only, while targets can also weigh IP reputation, cookie history, JavaScript execution, and behavior patterns. Does scrapy-redis prevent duplicate requests across several workers?Its Redis-backed duplicate filter shares request fingerprints across workers using the same key prefix and settings, so no worker schedules the same URL twice. That's shared deduplication, not a guarantee of exactly-once processing if a worker fails mid-request. What's the difference between Scrapyd and scrapyd-client?Scrapyd is the server that stores uploaded project versions and runs scheduled jobs through its own daemon and API. scrapyd-client is the local tool that packages and uploads a project to that server, without running crawls itself. Is it legal to use these extensions to scrape a website?Scraping publicly accessible pages is generally lawful in most jurisdictions. A site's terms of service, the data involved, and how you use the result can all change that. Check a target's robots.txt and terms before scraping it, and treat personal data with extra care under regulations like GDPR. ## Summary Integration point, current compatibility, and failure boundary matter more here than how many packages a list includes. Identify the missing output your crawl needs first. Verify the candidate package against your project's Scrapy version before writing a line of spider code. Add one integration at a time rather than stacking several untested changes together. The default path on Scrapy 2.18 in 2026 stays plain Scrapy first. Reach for scrapy-playwright only when a request genuinely needs browser rendering. Add a dedicated queue or deployment component only once the crawl needs shared state or a server to run on. Most of the nine packages above solve a narrow problem well. None of them substitute for checking that the version you pinned still imports on the Scrapy release you're running. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [Install by Missing Layer: Rendering, Fingerprints, Proxies, or Queue](#install-by-missing-layer-rendering-fingerprints-proxies-or-queue) - [Add-ons, Middlewares, Download Handlers, Schedulers, and Services Own Different Settings](#add-ons-middlewares-download-handlers-schedulers-and-services-own-different-settings) - [Project Setup](#project-setup) - [1. scrapy-playwright: Best for Selective JavaScript Rendering](#1-scrapy-playwright-best-for-selective-javascript-rendering) - [2. scrapy-selenium4: Best for Maintaining Selenium-Based Scrapy Crawlers](#2-scrapy-selenium4-best-for-maintaining-selenium-based-scrapy-crawlers) - [3. scrapy-splash: Best for Maintaining a Pinned Splash Stack](#3-scrapy-splash-best-for-maintaining-a-pinned-splash-stack) - [4. scrapy-impersonate: Best for Browser-Like TLS Fingerprints Without a Browser](#4-scrapy-impersonate-best-for-browser-like-tls-fingerprints-without-a-browser) - [5. scrapy-rotating-proxies: Best for a Small Self-Managed Proxy List](#5-scrapy-rotating-proxies-best-for-a-small-self-managed-proxy-list) - [6. scrapy-poet: Best for Reusable Extraction Across Many Sites](#6-scrapy-poet-best-for-reusable-extraction-across-many-sites) - [7. scrapy-redis: Best for a Shared Queue and Duplicate Filter](#7-scrapy-redis-best-for-a-shared-queue-and-duplicate-filter) - [8. Scrapyd: Best for Running Long-Lived Scrapy Jobs](#8-scrapyd-best-for-running-long-lived-scrapy-jobs) - [9. scrapyd-client: Best for Packaging and Uploading Scrapy Projects](#9-scrapyd-client-best-for-packaging-and-uploading-scrapy-projects) - [Which Scrapy Package Recommendations Are Stale on Scrapy 2.18?](#which-scrapy-package-recommendations-are-stale-on-scrapy-2-18) - [Power-up with Scrapfly](#power-up-with-scrapfly) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ python nodejs ### How to use Headless Chrome Extensions for Web Scraping In this article, we'll explore different useful Chrome extensions for web scraping. We'll also explain how to install Ch... ](https://scrapfly.io/blog/posts/how-to-use-browser-extensions-with-playwright-puppeteer-and-selenium) [ ### TLS Cipher Suites in Web Scraping Learn how TLS cipher suites affect both transport security and scraper detection, with practical JA3/JA4 guidance, refer... ](https://scrapfly.io/blog/posts/tls-cipher-suites-in-web-scraping) [ blocking proxies ### The Complete Guide To Using Proxies For Web Scraping Introduction to proxy usage in web scraping. What types of proxies are there? How to evaluate proxy providers and avoid ... ](https://scrapfly.io/blog/posts/introduction-to-proxies-in-web-scraping) ## Related Questions - [ Q How to add headers to every or some scrapy requests? ](https://scrapfly.io/blog/answers/how-to-add-headers-to-every-or-some-scrapy-requests) - [ Q What are private proxies and how are they used in scraping? ](https://scrapfly.io/blog/answers/what-are-private-proxies-compared-to-shared) - [ Q What are scrapy middlewares and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-middlewares-and-how-to-use-them) Run headless browsers at scale, **1,000 free credits** [Start Free](https://scrapfly.io/register)