[Blog](https://scrapfly.io/blog) / [headless-browser](https://scrapfly.io/blog/tag/headless-browser) / [9 Scrapy Extensions and Middlewares for 2026](https://scrapfly.io/blog/posts/best-scrapy-extensions-middlewares) # 9 Scrapy Extensions and Middlewares for 2026 by [Mayada Shaaban](https://scrapfly.io/blog/author/mayada-shaaban-90143e67) Aug 31, 2026 17 min read [\#headless-browser](https://scrapfly.io/blog/tag/headless-browser) [\#proxies](https://scrapfly.io/blog/tag/proxies) [\#python](https://scrapfly.io/blog/tag/python) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares&text=9%20Scrapy%20Extensions%20and%20Middlewares%20for%202026 "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-scrapy-extensions-middlewares) Scrapy 2.18 changed enough of the startup and spider-middleware contracts that several familiar package recommendations now fail before the first useful response. A package that imported cleanly on Scrapy 2.12 can raise a `TypeError` at crawler startup on 2.18. This list covers nine packages by job: rendering, TLS fingerprints, proxy rotation, reusable extraction, a shared crawl queue, and deployment. scrapy-selenium4 and scrapy-splash appear as maintenance choices, not picks for a new build. [Web Scraping With Scrapy: The Complete Guide in 2026Build and run a Scrapy 2.18 project with async start(), pagination, selector tests, item validation, pipelines, and JSON Lines export.](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) ## Key Takeaways - **Install by gap.** Add whatever component fills the missing layer. - **scrapy-playwright leads rendering.** The default pick, tested clean on Scrapy 2.18.0. - **Selenium and Splash stay pinned.** Maintenance-only choices, not new-build defaults. - **scrapy-impersonate matches TLS.** Spoofs fingerprints without JavaScript, not a bypass. - **scrapy-redis and scrapy-poet split jobs.** Shared queue versus reusable extraction. - **Scrapyd runs, scrapyd-client ships.** The server runs jobs, the client deploys versions. - **A PyPI date proves nothing.** scrapy-deltafetch still fails at Scrapy 2.18 startup. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## Which Scrapy Extension or Middleware Should You Install? Install scrapy-playwright for selective browser rendering, scrapy-impersonate for HTTP fingerprint control, scrapy-redis for a shared queue and duplicate filter, and the Scrapyd tools for deployment. These nine components form a stack ordered by workflow, not nine substitutes for the same job. Each row names the reader problem it solves and the boundary it was tested against. | \# | Component | Reader job | Primary output | Tested boundary | |---|---|---|---|---| | 1 | scrapy-playwright | Render JavaScript | Rendered response | 2.18.0 + 0.0.48 | | 2 | scrapy-selenium4 | Keep Selenium flow | Selenium response | 2.18.0 + warning | | 3 | scrapy-splash | Keep Splash flow | HTTP or Lua render | 2.12.0 pinned | | 4 | scrapy-impersonate | Match TLS profile | HTTP response | 2.18.0 + 1.9.0 | | 5 | scrapy-rotating-proxies | Rotate raw proxies | Proxy lifecycle | 2.18.0 + 0.6.2 | | 6 | scrapy-poet | Reuse extraction | Injected page object | 2.18.0 + 0.27.2 | | 7 | scrapy-redis | Share crawl state | Redis queue and set | 2.18.0 + 0.9.1 | | 8 | Scrapyd | Run crawl jobs | Job API and logs | 2.18.0 + 1.6.0 | | 9 | scrapyd-client | Ship project builds | Uploaded project version | 2.18.0 + 2.0.3 | Install the ones you need with pinned versions: bash```bash pip install "scrapy==2.18.0" "scrapy-playwright==0.0.48" "scrapy-impersonate==1.9.0" \ "scrapy-rotating-proxies==0.6.2" "scrapy-poet==0.27.2" "scrapy-redis==0.9.1" \ "scrapyd==1.6.0" "scrapyd-client==2.0.3" ``` scrapy-selenium4 and scrapy-splash are deliberately absent: see entries 2 and 3 for the pinned stacks they need. Read the table top to bottom by what your crawl is missing today, not by package popularity. A crawl that only needs raw HTTP has no reason to add a browser handler, and a single-machine crawl has no reason to add scrapy-redis. Plain Scrapy covers the request and response cycle these packages plug into. ## How Do Scrapy Add-ons, Middlewares, Download Handlers, Schedulers, and Services Differ? Each integration point owns a different part of the crawl lifecycle, so its class belongs under the setting that package documents, not wherever it happens to import cleanly. A class registered under the wrong setting usually raises nothing at all, which is why this table comes before the entries. | Integration point | Setting or boundary | What it changes | |---|---|---| | Add-on | `ADDONS` | Several settings as one unit | | Extension | `EXTENSIONS` | Signals, stats, and monitoring | | Downloader middleware | `DOWNLOADER_MIDDLEWARES` | Requests and responses | | Spider middleware | `SPIDER_MIDDLEWARES` | Spider input and output | | Download handler | `DOWNLOAD_HANDLERS` | Network transport | | Scheduler | `SCHEDULER` | Request queue and priority | | Service or CLI | Separate process | Running or exposing spiders | scrapy-playwright and scrapy-impersonate both swap in a download handler, so they own network transport per scheme rather than touching every request through a middleware hook. scrapy-redis replaces the scheduler and duplicate-filter classes instead, and Scrapyd runs as a separate service with its own process and API. Getting a package's class into the wrong setting is a fast way to have it silently do nothing. ## 1. scrapy-playwright: Best for Selective JavaScript Rendering scrapy-playwright 0.0.48 is the default choice when selected Scrapy requests need a current Playwright browser. The tested boundary is Scrapy 2.18.0, scrapy-playwright 0.0.48, and Playwright 1.62.0, running Chrome for Testing 151.0.7922.34 as the Chromium build. It works as a download handler activated per request through `meta={"playwright": True}`, so plain HTTP requests in the same spider skip the browser entirely. Inspect the initial HTML and the page's background requests before reaching for this handler. Many pages that look JavaScript-rendered actually expose the same data through a plain XHR call a normal request can read directly. python```python import scrapy from scrapy.crawler import CrawlerProcess SETTINGS = { "DOWNLOAD_HANDLERS": { "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", }, "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", } class ProductsSpider(scrapy.Spider): name = "products_playwright" async def start(self): yield scrapy.Request( "https://web-scraping.dev/products", meta={"playwright": True}, ) def parse(self, response): cards = response.css(".product") for card in cards: yield { "title": card.css("h3 a::text").get(default="").strip(), "price": card.css(".price::text").get(default="").strip(), } if __name__ == "__main__": process = CrawlerProcess(SETTINGS) process.crawl(ProductsSpider) process.start() ``` This spider requests `web-scraping.dev/products` through the Playwright download handler, waits for the rendered response, and yields the title and price scraped from each product card. This run against `web-scraping.dev/products` returned five items through the rendered response and normal Scrapy selectors: json```json [ {"title": "Box of Chocolate Candy", "price": "24.99"}, {"title": "Dark Red Energy Potion", "price": "4.99"}, {"title": "Teal Energy Potion", "price": "4.99"}, {"title": "Red Energy Potion", "price": "4.99"}, {"title": "Blue Energy Potion", "price": "4.99"} ] ``` Browser requests consume more CPU and memory per page than plain HTTP, and running a browser at all does not guarantee anti-bot access on a protected target. For a full walkthrough of waits, browser contexts, and page lifecycle, see [the dedicated scrapy-playwright guide](https://scrapfly.io/blog/posts/web-scraping-dynamic-websites-with-scrapy-playwright). One boundary sits outside the scrapy-playwright handler entirely: direct Playwright can connect to a [Cloud Browser](https://scrapfly.io/products/cloud-browser-api) session over CDP once operating browser sessions becomes its own infrastructure problem, separate from the scraping logic. A Playwright 1.62 script connecting over CDP reached `https://web-scraping.dev/products`, returned HTTP 200, and counted five product cards. Grab the session's existing context with `browser.contexts[0]`. Calling `browser.new_page()` against a remote session raises a target error; `context.new_page()` on that context is fine. That's a direct Playwright-to-[Cloud Browser](https://scrapfly.io/docs/cloud-browser-api/getting-started) connection over CDP, not a scrapy-playwright download-handler configuration. ## 2. scrapy-selenium4: Best for Maintaining Selenium-Based Scrapy Crawlers Use `scrapy-selenium4==1.0.0` only when an existing project already depends on Selenium WebDriver behavior inside Scrapy. It is a maintenance choice, not a starting point for a new browser integration. This is a different package from the original `scrapy-selenium==0.0.7`, which lives at a different repository and targets Selenium 3. The reproduced matrix here is Scrapy 2.18.0, Selenium 4.47.0, and Chrome and ChromeDriver both on major version 151, resolved automatically by Selenium Manager rather than pinned by hand. The package's middleware still uses the two-argument `process_request(self, request, spider)` signature Scrapy is phasing out. Scrapy raises a `ScrapyDeprecationWarning` once, at crawler startup, when it registers that method and finds the `spider` parameter still required. Its repository, a fork at `zhouyangchao/scrapy-selenium`, last committed in September 2024 and carries a single star with issues disabled. Browser work inside the middleware also runs synchronously, blocking the reactor for each page load. For a new browser integration, use scrapy-playwright instead. See [the existing Scrapy Selenium guide](https://scrapfly.io/blog/posts/web-scraping-dynamic-web-pages-with-scrapy-selenium) for the working setup this maintenance path assumes. ## 3. scrapy-splash: Best for Maintaining a Pinned Splash Stack Keep scrapy-splash only for an existing simple-render or Lua workflow that can stay on a frozen stack. `scrapy-splash==0.11.1` fails to import on Scrapy 2.18: it imports `canonicalize_url` from `scrapy.utils.url`, and current Scrapy no longer exposes that function there (it now lives in `w3lib.url`). The reproduced working baseline is Python 3.12, Scrapy 2.12.0, Twisted 24.11.0, scrapy-splash 0.11.1, and Splash 3.5. Splash itself is a separate HTTP rendering service, not a Python library that runs in-process, and its 3.5 server ships a legacy browser engine that predates current Chromium-based rendering. Under that pinned stack the import succeeds, and a `SplashRequest` against a local Splash service returned a fully rendered page. Route new browser work to scrapy-playwright instead. See [the dedicated Scrapy Splash guide](https://scrapfly.io/blog/posts/web-scraping-with-scrapy-splash) for the full pinned setup and Lua examples. ## 4. scrapy-impersonate: Best for Browser-Like TLS Fingerprints Without a Browser scrapy-impersonate 1.9.0 replaces Scrapy's HTTP download handler with curl\_cffi, so a request can present a browser-like TLS and HTTP fingerprint without running JavaScript. The tested boundary is Scrapy 2.18.0 with `impersonate="chrome146"`, a real curl\_cffi profile in this release, which returned HTTP 200 and a non-empty JA3 hash from a public TLS fixture. Leave `USER_AGENT` empty so curl\_cffi picks a value that matches the impersonation profile automatically, and keep the asyncio reactor active for the handler to load. Redirect handling is worth calling out: this handler has no bare `allow_redirects` request argument the way `requests` or `curl_cffi` do. Redirects stay under Scrapy's own `RedirectMiddleware` unless overridden through `meta["impersonate_args"]`. python```python import json import scrapy class ImpersonateSpider(scrapy.Spider): name = "impersonate_test" custom_settings = { "DOWNLOAD_HANDLERS": { "http": "scrapy_impersonate.ImpersonateDownloadHandler", "https": "scrapy_impersonate.ImpersonateDownloadHandler", }, "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", "USER_AGENT": "", } async def start(self): meta = {"impersonate": "chrome146"} yield scrapy.Request("https://tls.peet.ws/api/all", meta=meta) def parse(self, response): data = json.loads(response.text) ja3_hash = data.get("tls", {}).get("ja3_hash") self.logger.info("STATUS=%s JA3_HASH=%s", response.status, ja3_hash) yield {"status": response.status, "ja3_hash": ja3_hash} ``` Two separate runs against the live fixture both returned status 200 with a populated hash, for example `ja3_hash: "5c9265d5eed9cdeb0cbd9c3fd2111388"` on one run. The hash itself varies between runs and targets, so treat it as a live signal to check, not a constant to hard-code. This changes HTTP transport signals only, not JavaScript execution, cookies earned through page interaction, or human behavior patterns. A matching JA3 hash does not identify every browser or clear every anti-bot check on its own. See [TLS fingerprint blocking](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-tls) for how targets use this signal alongside others. ## 5. scrapy-rotating-proxies: Best for a Small Self-Managed Proxy List scrapy-rotating-proxies 0.6.2 can still assign and cool down members of a raw proxy list, but it needs a pinned compatibility check and a target-specific ban policy before you rely on it. The release was uploaded to PyPI on May 25, 2019, and its packaging still declares an unpinned dependency on the old `typing` backport. That backport doesn't block installation on current Python, contrary to what its age might suggest. It installed cleanly alongside Scrapy 2.18.0 on both Python 3.13 and 3.14 in a fresh check, but not via the current `typing` release. `typing 3.10.0.0` declares `Requires-Python: <3.5`, so pip skips it and resolves `typing 3.7.4.3` instead, which carries no upper bound, then builds it from source. The dependency is satisfied by a 2020 release that pip picked silently. The real caveat is packaging hygiene, not a hard install block. Under Scrapy 2.18 the middleware loaded correctly and rotated requests across both members of a two-proxy test fixture. That run also logged five `ScrapyDeprecationWarning`s, one per hook on both middlewares, because Scrapy will stop passing the `spider` argument in a future release. A 403 or 429 response describes what the target returned, not automatically a dead proxy, so pair this package with a policy that fits your target rather than trusting its defaults. A minimal setup adds `rotating_proxies.middlewares.RotatingProxyMiddleware` at priority 610 and `BanDetectionMiddleware` at 620 under `DOWNLOADER_MIDDLEWARES`, with `ROTATING_PROXY_LIST` pointed at your own proxy addresses. For the full proxy-list lifecycle and ban-detection tuning, see [proxy rotation strategies](https://scrapfly.io/blog/posts/how-to-rotate-proxies-in-web-scraping). This entry does not change TLS or browser fingerprints. It only changes which upstream IP a request leaves from. ## 6. scrapy-poet: Best for Reusable Extraction Across Many Sites scrapy-poet 0.27.2 separates extraction into injected page objects, so one spider can reuse traversal logic across sites with different page layouts. Activate it with `ADDONS = {"scrapy_poet.Addon": 300}` on Scrapy 2.10 and newer. On Scrapy 2.18 that add-on injected a `WebPage`-based object into a callback and returned the expected record from a live page. python```python import scrapy from web_poet import WebPage from scrapy.crawler import CrawlerProcess class ProductPage(WebPage): async def to_item(self) -> dict: return { "name": self.css("h3.product-title::text").get("").strip(), "price": self.css("span.product-price::text").get("").strip(), } class PoetSpider(scrapy.Spider): name = "poet_test" custom_settings = {"ADDONS": {"scrapy_poet.Addon": 300}} async def start(self): yield scrapy.Request("https://web-scraping.dev/product/1", callback=self.parse) async def parse(self, response, page: ProductPage): item = await page.to_item() self.logger.info("EXTRACTED_ITEM=%r", item) yield item if __name__ == "__main__": process = CrawlerProcess() process.crawl(PoetSpider) process.start() ``` The run against `web-scraping.dev/product/1` logged `EXTRACTED_ITEM={'name': 'Box of Chocolate Candy', 'price': '$9.99'}`, and the crawl stats confirmed the injector actually constructed `ProductPage` rather than passing through a plain response. The output here is an independently testable extraction object, not faster downloads or a shared crawl queue. Those are separate jobs further down this list. Scrapfly #### Need a cloud browser for scraping? Run headless browsers at scale with Scrapfly Cloud Browser — no infrastructure to manage. [Try Free →](https://scrapfly.io/register)## 7. scrapy-redis: Best for a Shared Queue and Duplicate Filter scrapy-redis 0.9.1 gives multiple Scrapy workers a Redis-backed request queue, duplicate filter, and optional item pipeline. On Scrapy 2.18, the priority queue round-tripped a `scrapy.Request` with its URL, priority, and metadata intact, and the Redis duplicate filter correctly rejected the same request fingerprint on its second check. python```python import os os.environ.setdefault("REDIS_URL", "redis://localhost:6379/0") REDIS_URL = os.environ["REDIS_URL"] SCHEDULER = "scrapy_redis.scheduler.Scheduler" DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter" SCHEDULER_PERSIST = True SCHEDULER_QUEUE_KEY = "myproject:%(spider)s:requests" SCHEDULER_DUPEFILTER_KEY = "myproject:%(spider)s:dupefilter" ``` Every worker needs to share the same Redis keys and project settings for queue and deduplication behavior to stay coherent between processes. A worker pointed at a different key prefix won't see the others' requests. This package supplies shared queue and fingerprint state, not an advanced crawl-frontier policy, worker deployment, or exactly-once processing guarantees. A request that a worker dequeues but crashes before finishing can still be reprocessed. [Concurrency vs ParallelismLearn the key differences between Concurrency and Parallelism and how to leverage them in Python and JavaScript to optimize performance in various computational tasks.](https://scrapfly.io/blog/posts/concurrency-vs-parallelism) ## 8. Scrapyd: Best for Running Long-Lived Scrapy Jobs Scrapyd 1.6.0 is the deployment target when Scrapy projects need a daemon and JSON API for uploaded versions, scheduled jobs, logs, and items. Running it beside Scrapy 2.18 and querying `daemonstatus.json` returned a live `{"status": "ok", ...}` response, and it accepted a project upload through its deploy endpoint without issue. Scrapyd is the server component only. It doesn't build or upload a project by itself. That's scrapyd-client's job, covered next. Keep production hardening, process supervision, and network access control out of a quick setup: Scrapyd's own docs cover running it behind a reverse proxy with authentication, and none of that belongs on a daemon exposed directly to the internet. ## 9. scrapyd-client: Best for Packaging and Uploading Scrapy Projects scrapyd-client 2.0.3 builds a Scrapy project artifact, uploads it through Scrapyd's `addversion.json` API, and can query or schedule the project it just deployed. With Scrapy 2.18 and Scrapyd 1.6.0 running locally, `scrapyd-client deploy` built the project and received a 200 response with `status: ok`. ini```ini # scrapy.cfg [deploy:local] url = ${SCRAPYD_URL} project = demo ``` bash```bash export SCRAPYD_URL=http://localhost:6800 scrapyd-client deploy local -p demo scrapyd-client projects -t local ``` The `scrapy.cfg` block names a deploy target and points it at the running Scrapyd instance. The commands below it export that URL, build and upload the project to `demo`, then list the projects registered on that target. Version 2.0.3 ships a unified `scrapyd-client` command with `deploy`, `targets`, `projects`, `schedule`, and `spiders` subcommands, alongside the older standalone `scrapyd-deploy`script that still works through the same code path. The legacy `-l` and `-L` list flags on `scrapyd-deploy` were removed in 2.0.0. Use `scrapyd-client projects` and `scrapyd-client spiders` instead. Environment-variable interpolation in `scrapy.cfg` works natively, so keep the URL out of the file and export it at deploy time rather than committing it. The output here is a versioned project available to Scrapyd, not a running crawl until something schedules it. ## Which Scrapy Package Recommendations Are Stale on Scrapy 2.18? A recent release date does not prove compatibility with Scrapy 2.18's asynchronous spider-output contract. Two packages that still show up in search results and generated answers fail outright once a project upgrades. | Package | Current finding | Safer route | |---|---|---| | scrapy-deltafetch 2.1.0 | Fails at crawler startup | Redis duplicate filter or native job state | | scrapy-selenium 0.0.7 | Selenium 3 constructor path | scrapy-selenium4 for maintenance | | scrapy-splash 0.11.1 | Import fails on Scrapy 2.18 | Pinned Scrapy 2.12 or Playwright | scrapy-deltafetch's spider middleware implements a plain synchronous `process_spider_output`, and Scrapy 2.18 requires spider middlewares to support asynchronous output. Loading it raises `TypeError: Middleware scrapy_deltafetch.middleware.DeltaFetch doesn't support asynchronous spider output` at crawler startup, before a single request goes out. scrapy-selenium 0.0.7 still instantiates its WebDriver with `executable_path` and `chrome_options` keyword arguments, which Selenium 4 removed. That's the constructor path scrapy-selenium4 exists to fix. These are observed behaviors on Scrapy 2.18, not judgments about whether a project is abandoned. Each row is a version, what it does when you load it, and where to go instead. ## Power-up with Scrapfly Every extension in this list still leaves rendering, fingerprinting, and proxy rotation as separate packages to install and monitor. Scrapfly's [Web Scraping API](https://scrapfly.io/products/web-scraping-api) is a single HTTP endpoint for collecting web data at scale, with a **98% success rate** across **130M+ proxies in 190+ countries**. - [Anti-Scraping Protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - automatically defeats Cloudflare, DataDome, PerimeterX, Akamai, and 90+ other bot systems. - [Smart proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) - residential and datacenter pools with country and ASN level geo-targeting. - [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - render SPAs and dynamic pages through real cloud browsers. - [Browser automation scenarios](https://scrapfly.io/docs/scrape-api/javascript-scenario) - scroll, click, fill forms, and wait for elements without managing a browser fleet. - [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - return pages as HTML, JSON, clean text, or LLM ready Markdown. - [Session management](https://scrapfly.io/docs/scrape-api/session) - keep cookies, headers, and IPs consistent across multi step flows. - [Smart caching](https://scrapfly.io/docs/scrape-api/getting-started#api_param_cache) - cache successful responses to cut cost on repeat scraping jobs. - [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Scrapy](https://scrapfly.io/docs/sdk/scrapy), and [no-code integrations](https://scrapfly.io/docs/integration/getting-started) including [Make](https://scrapfly.io/integration/make), [n8n](https://scrapfly.io/integration/n8n), [Zapier](https://scrapfly.io/integration/zapier), [LangChain](https://scrapfly.io/integration/langchain), and [LlamaIndex](https://scrapfly.io/integration/llamaindex). The Scrapy SDK drops into the same `DOWNLOAD_HANDLERS` setting this article's packages use, so switching to Scrapfly doesn't mean rewriting existing spider logic. ### Web Scraping API Scrape any website with our powerful API. Anti-bot bypass, JavaScript rendering, and rotating proxies built-in. [Try Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started) ## FAQ Can scrapy-playwright, scrapy-impersonate, and scrapy-redis run in the same project?Yes, but not by accident. scrapy-redis replaces the scheduler and duplicate filter, while scrapy-playwright and scrapy-impersonate are both download handlers competing for the same `http` and `https` keys. Only one can own a scheme, so route the other per request through its own `meta` key or a separate scheme entry. Does scrapy-impersonate bypass every anti-bot system?No. It changes transport-level fingerprints like TLS and HTTP/2 signals, while targets can also evaluate IP reputation, cookie history, JavaScript execution, and behavioral signals that transport impersonation doesn't touch. Does scrapy-redis prevent duplicate requests across several workers?Its Redis-backed duplicate filter can share request fingerprints across workers that use the same key prefix and settings, which stops the same URL from being scheduled twice. That's shared deduplication, not a guarantee of exactly-once processing if a worker fails mid-request. What's the difference between Scrapyd and scrapyd-client?Scrapyd is the server that stores uploaded project versions and runs scheduled jobs through its own daemon and API. scrapyd-client is the local tool that packages a project and uploads it to that server. It doesn't run crawls itself. Is it legal to use these extensions to scrape a website?Scraping publicly accessible pages is generally lawful in most jurisdictions, but a site's terms of service, the data involved, and how you use the result can all change that. Check a target's robots.txt and terms before scraping it, and treat personal data with extra care under regulations like GDPR. ## Summary Integration point, current compatibility, and failure boundary decide whether a package works on 2.18, not how many packages a list includes. Identify the missing output your crawl needs first, verify the candidate package against your project's Scrapy version before writing a line of spider code, and add one integration at a time rather than stacking several untested changes together. The default path on Scrapy 2.18 in 2026 stays plain Scrapy first. Reach for scrapy-playwright only when a request genuinely needs browser rendering, and add a dedicated queue or deployment component only once the crawl needs shared state or a server to run on. Most of the nine packages above solve a narrow problem well. None of them substitute for checking that the version you pinned still imports on the Scrapy release you're running. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [Which Scrapy Extension or Middleware Should You Install?](#which-scrapy-extension-or-middleware-should-you-install) - [How Do Scrapy Add-ons, Middlewares, Download Handlers, Schedulers, and Services Differ?](#how-do-scrapy-add-ons-middlewares-download-handlers-schedulers-and-services-differ) - [1. scrapy-playwright: Best for Selective JavaScript Rendering](#1-scrapy-playwright-best-for-selective-javascript-rendering) - [2. scrapy-selenium4: Best for Maintaining Selenium-Based Scrapy Crawlers](#2-scrapy-selenium4-best-for-maintaining-selenium-based-scrapy-crawlers) - [3. scrapy-splash: Best for Maintaining a Pinned Splash Stack](#3-scrapy-splash-best-for-maintaining-a-pinned-splash-stack) - [4. scrapy-impersonate: Best for Browser-Like TLS Fingerprints Without a Browser](#4-scrapy-impersonate-best-for-browser-like-tls-fingerprints-without-a-browser) - [5. scrapy-rotating-proxies: Best for a Small Self-Managed Proxy List](#5-scrapy-rotating-proxies-best-for-a-small-self-managed-proxy-list) - [6. scrapy-poet: Best for Reusable Extraction Across Many Sites](#6-scrapy-poet-best-for-reusable-extraction-across-many-sites) - [7. scrapy-redis: Best for a Shared Queue and Duplicate Filter](#7-scrapy-redis-best-for-a-shared-queue-and-duplicate-filter) - [8. Scrapyd: Best for Running Long-Lived Scrapy Jobs](#8-scrapyd-best-for-running-long-lived-scrapy-jobs) - [9. scrapyd-client: Best for Packaging and Uploading Scrapy Projects](#9-scrapyd-client-best-for-packaging-and-uploading-scrapy-projects) - [Which Scrapy Package Recommendations Are Stale on Scrapy 2.18?](#which-scrapy-package-recommendations-are-stale-on-scrapy-2-18) - [Power-up with Scrapfly](#power-up-with-scrapfly) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ python nodejs ### How to use Headless Chrome Extensions for Web Scraping In this article, we'll explore different useful Chrome extensions for web scraping. We'll also explain how to install Ch... ](https://scrapfly.io/blog/posts/how-to-use-browser-extensions-with-playwright-puppeteer-and-selenium) [ ### TLS Cipher Suites in Web Scraping Learn how TLS cipher suites affect both transport security and scraper detection, with practical JA3/JA4 guidance, refer... ](https://scrapfly.io/blog/posts/tls-cipher-suites-in-web-scraping) [ blocking proxies ### The Complete Guide To Using Proxies For Web Scraping Introduction to proxy usage in web scraping. What types of proxies are there? How to evaluate proxy providers and avoid ... ](https://scrapfly.io/blog/posts/introduction-to-proxies-in-web-scraping) ## Related Questions - [ Q How to add headers to every or some scrapy requests? ](https://scrapfly.io/blog/answers/how-to-add-headers-to-every-or-some-scrapy-requests) - [ Q What are private proxies and how are they used in scraping? ](https://scrapfly.io/blog/answers/what-are-private-proxies-compared-to-shared) - [ Q What are scrapy middlewares and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-middlewares-and-how-to-use-them) Run headless browsers at scale, **1,000 free credits** [Start Free](https://scrapfly.io/register)