[Blog](https://scrapfly.io/blog) / [frameworks](https://scrapfly.io/blog/tag/frameworks) / [Scrapy Splash Guide: Setup, Lua, and Compatibility](https://scrapfly.io/blog/posts/web-scraping-with-scrapy-splash) # Scrapy Splash Guide: Setup, Lua, and Compatibility by [Mazen Ramadan](https://scrapfly.io/blog/author/mazen) Aug 31, 2026 14 min read [\#frameworks](https://scrapfly.io/blog/tag/frameworks) [\#headless-browser](https://scrapfly.io/blog/tag/headless-browser) [\#python](https://scrapfly.io/blog/tag/python) [\#scrapy](https://scrapfly.io/blog/tag/scrapy) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy-splash "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy-splash&text=Scrapy%20Splash%20Guide%3A%20Setup%2C%20Lua%2C%20and%20Compatibility "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy-splash "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy-splash) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy-splash) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy-splash) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy-splash) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy-splash) Scrapy Splash still starts. It still renders pages and logs into forms on the right versions. But install `scrapy-splash` against a current Scrapy release, 2.18.0 today, and the crawl fails before it starts, because the package imports a URL helper that Scrapy already removed. This guide freezes a stack that still runs: Python 3.12, Scrapy 2.12.0, Twisted 24.11.0, `scrapy-splash` 0.11.1, and Splash 3.5. It runs two bounded jobs on that stack, a page render and a form login, then reports one scroll test that held at its starting count without pinning down why. [Web Scraping With Scrapy: The Complete Guide in 2026Build and run a Scrapy 2.18 project with async start(), pagination, selector tests, item validation, pipelines, and JSON Lines export.](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) ## Key Takeaways - **`scrapy-splash` 0.11.1 fails to import on Scrapy 2.16.0 and later.** It imports `canonicalize_url` from a Scrapy module that no longer exports it, so the crawl stops before it starts, including on the current 2.18.0 release. - **This guide runs a frozen compatibility stack.** Python 3.12, Scrapy 2.12.0, Twisted 24.11.0, `scrapy-splash` 0.11.1, and Splash 3.5. - **Use Splash to maintain simple HTTP and Lua rendering flows.** It is not a current browser default for a new project. - **In a bounded test, Splash 3.5 held at the initial 10 records.** On this pinned stack, the testimonials page is not a working infinite-scroll example. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## Does Scrapy Splash still work in 2026? Splash still supports existing, simple rendering flows. But `scrapy-splash` 0.11.1 does not import on Scrapy 2.16.0 or later, including the current 2.18.0 release. Splash 3.5 server is legacy infrastructure now not a growing project. | Component | Version in this guide | Role | Limit | |---|---|---|---| | Python | 3.12 | Client runtime | Keep this pin with the full stack | | Scrapy | 2.12.0 | Crawl framework | Newer Scrapy releases cross breaking internal API changes | | Twisted | 24.11.0 | Scrapy networking dependency | Twisted 26.4.0 failed against Scrapy 2.12.0 at the removed `_setAcceptableProtocols` internal. Keep this tested pin | | scrapy-splash | 0.11.1 | Scrapy integration | Imports fail on Scrapy 2.16.0 and later, including 2.18.0 | | Splash server | 3.5 | Rendering service | Released in 2020 and runs an older browser engine | | Docker image | `scrapinghub/splash:3.5` | Server distribution | The tested image was Linux amd64. This guide validated no ARM image | Use this stack to maintain an existing Splash crawler. For a new Scrapy browser project use Playwright and a current Chromium engine. That status table drives every version pin in the setup that follows. ## How do you run Splash 3.5 with Docker? ### How do you start the pinned Splash image? You need [Docker](https://docs.docker.com/get-started/get-docker/) installed. Pull the exact tag this guide tested, keep it off the public network, and confirm it answers before pointing Scrapy at it: bash```bash docker pull scrapinghub/splash:3.5 docker run -d --name splash-pinned -p 127.0.0.1:8050:8050 scrapinghub/splash:3.5 curl http://127.0.0.1:8050/_ping # stop it when you're done testing docker stop splash-pinned ``` The `-p 127.0.0.1:8050:8050` flag binds Splash to loopback only, so nothing outside your machine can reach the render API. A successful `/_ping` response confirms the container answers, and `docker stop` shuts it down when you're finished. ### What does the Splash 3.5 runtime limit? Splash 3.5 has real limits worth knowing before you build on it: - Splash 3.5 shipped in 2020 and runs an older browser engine than any current browser, per the official Splash change log. - A successful `/_ping` response proves the service is running, not that a modern JavaScript-heavy page renders correctly. - The image this guide tested was Linux amd64. This guide validated no ARM image. - Keep the HTTP API on loopback or behind an access-controlled private service. The API accepts arbitrary target URLs and Lua scripts, so a publicly reachable instance lets anyone run scripts through your server. With the container running and locked to loopback, the next step is pinning the Python side to match it. ## How do you install the pinned scrapy-splash stack? ### Which Python packages must stay pinned together? Create a Python 3.12 environment, install the three packages at exact versions, and confirm what landed before wiring `scrapy-splash` into a real project: bash```bash # Python 3.12, matched to the pinned stack python3.12 -m venv splash-env source splash-env/bin/activate pip install scrapy==2.12.0 twisted==24.11.0 scrapy-splash==0.11.1 python -c "import scrapy, twisted; print('scrapy', scrapy.__version__); print('twisted', twisted.__version__)" pip show scrapy-splash | grep -i "^Version:" ``` The version print is the check that matters. A stray `pip install --upgrade` later is how one of these pins quietly moves and breaks the crawl. ### Why does scrapy-splash fail on Scrapy 2.16.0 and later? `scrapy-splash` 0.11.1 imports `canonicalize_url` from `scrapy.utils.url` inside its dupefilter module. Scrapy 2.16.0 removed that function and the current 2.18.0 release still does not export it, so the import fails before your spider starts. [scrapy-splash issue 332](https://github.com/scrapy-plugins/scrapy-splash/issues/332) tracks the break upstream. Don't patch the installed package files or monkey-patch Scrapy internals to force the import through. Pin the versions above instead, since that is the exact boundary this guide tested. With the pins confirmed, the next step is wiring `scrapy-splash` into `settings.py`. ## How do you configure scrapy-splash for the pinned Scrapy project? ### Which Splash middlewares and fingerprinter should settings.py use? Add this block to `settings.py`. It's the same configuration this guide ran against the pinned Splash container: python```python # settings.py SPLASH_URL = "http://127.0.0.1:8050" DOWNLOADER_MIDDLEWARES = { "scrapy_splash.SplashCookiesMiddleware": 723, "scrapy_splash.SplashMiddleware": 725, "scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": 810, } SPIDER_MIDDLEWARES = { "scrapy_splash.SplashDeduplicateArgsMiddleware": 100, } # replaces the legacy SplashAwareDupeFilter for request deduplication REQUEST_FINGERPRINTER_CLASS = "scrapy_splash.SplashRequestFingerprinter" ``` Two downloader middlewares, one spider middleware, and `SplashRequestFingerprinter` instead of a custom dupefilter. `scrapy-splash` 0.11.1 still ships the older `SplashAwareDupeFilter` and `SplashAwareFSCacheStorage` classes and this project skips both. Scrapy's own request fingerprinting replaces the dupefilter and its HTTP cache replaces the cache storage class. The pinned integration still reaches into deprecated Scrapy downloader internals to route requests through Splash. That's why this guide freezes the whole stack instead of claiming it works on any Scrapy version you happen to run. ## How do you render one page with SplashRequest? ### How does a Scrapy request reach the Splash HTTP API? This spider sends a bounded `render.html` request to the testimonials test page and parses the result like any other Scrapy response: python```python import scrapy from scrapy.crawler import CrawlerProcess from scrapy_splash import SplashRequest class ReviewsSpider(scrapy.Spider): name = "reviews" custom_settings = { "SPLASH_URL": "http://127.0.0.1:8050", "DOWNLOADER_MIDDLEWARES": { "scrapy_splash.SplashCookiesMiddleware": 723, "scrapy_splash.SplashMiddleware": 725, "scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": 810, }, "SPIDER_MIDDLEWARES": {"scrapy_splash.SplashDeduplicateArgsMiddleware": 100}, "REQUEST_FINGERPRINTER_CLASS": "scrapy_splash.SplashRequestFingerprinter", } def start_requests(self): url = "https://web-scraping.dev/testimonials" yield SplashRequest( url=url, callback=self.parse, endpoint="render.html", args={"wait": 2}, ) def parse(self, response): title = response.css("title::text").get() reviews = response.css("div.testimonial") print("title:", title) print("review count:", len(reviews)) for review in reviews[:3]: print({ "rate": len(review.css("span.rating > svg").getall()), "text": review.css("p.text::text").get(), }) process = CrawlerProcess() process.crawl(ReviewsSpider) process.start() ``` Running this against the pinned Splash container prints the page title, the review count, and the first three parsed reviews: text```text title: web-scraping.dev testimonials review count: 10 {'rate': 5, 'text': "We've been using this utility for years - awesome service!"} {'rate': 5, 'text': 'This Python app simplified my workflow significantly. Highly recommended.'} {'rate': 4, 'text': 'Had a few issues at first, but their support team is top-notch!'} ``` This `start_requests()` form matches Scrapy 2.12's startup path. Scrapy deprecated `start_requests()` in 2.13.0 and removed it in 2.16.0 for an async `start()` method, so don't carry this pattern into a tutorial written against a newer release. ### What is the difference between the request to Splash and the request to the target? Two separate HTTP hops happen behind one `SplashRequest`: - Scrapy sends a request to the Splash service, not directly to the target site. - Splash then requests the target itself, using the render endpoint or Lua script arguments you provided. - For standard render endpoints, `SplashRequest` maps your target URL, method, and body straight into Splash's request arguments. - For the `/execute` endpoint, the Lua script owns any explicit target method, body, and header forwarding. That two-hop model is also why Lua scripts, covered next, need their own argument handling instead of relying on Scrapy's request object. ## How do you run Lua scripts through scrapy-splash? ### How does SplashRequest call the execute endpoint? Splash's `/execute` endpoint runs a Lua script with access to both Splash functions and the page's JavaScript environment. Pass arguments through `splash.args` rather than interpolating strings into the script: python```python import scrapy from scrapy_splash import SplashRequest title_script = """ function main(splash, args) splash:go(args.url) splash:wait(args.wait) return { html = splash:html(), title = splash:evaljs("document.title"), } end """ class TitleSpider(scrapy.Spider): name = "title" def start_requests(self): yield SplashRequest( url="https://web-scraping.dev/testimonials", callback=self.parse, endpoint="execute", args={"lua_source": title_script, "wait": 2}, ) def parse(self, response): print("page title from Lua:", response.data["title"]) ``` Running this prints `page title from Lua: web-scraping.dev testimonials`. The script reads `args.url` and `args.wait` from `splash.args` instead of hardcoding them, so the same Lua source works across different target pages. ### How do you forward a target POST from a Splash Lua script? A Lua script can also set the request method, body, and headers directly through `splash:go`. This fragment is a conceptual pattern, not a tested endpoint call, and it stays separate from the working login flow below: lua```lua function main(splash, args) splash:go{ args.url, http_method=args.http_method, body=args.body, headers=args.headers, } return { html = splash:html() } end ``` Don't assume every target accepts the same body encoding. Confirm the target's expected content type before reusing this pattern. Scrapfly #### Need a cloud browser for scraping? Run headless browsers at scale with Scrapfly Cloud Browser — no infrastructure to manage. [Try Free →](https://scrapfly.io/register)## How do you submit a login form with Scrapy Splash? ### How do Splash form fill and submit work together? Splash's `form:fill` and `submit` methods work against the page's actual form element, which is more reliable than clicking individual fields with JavaScript. This example logs into the public test login page and checks for the authenticated marker before returning: python```python import scrapy from scrapy_splash import SplashRequest login_script = """ function main(splash, args) splash:go(args.url) splash:wait(1.0) local accept_cookies = splash:jsfunc([[ function () { var btn = document.querySelector("#cookie-ok"); if (btn) { btn.click(); } } ]]) accept_cookies() splash:wait(0.5) local form = splash:select("form") form:fill({username=args.username, password=args.password}) splash:wait(0.3) form:submit() local deadline = 8 local waited = 0 local step = 0.5 local logged_in = false while waited < deadline do splash:wait(step) waited = waited + step if splash:select("#secret-message") then logged_in = true break end end return { html = splash:html(), logged_in = logged_in, } end """ class LoginSpider(scrapy.Spider): name = "login" def start_requests(self): yield SplashRequest( url="https://web-scraping.dev/login", callback=self.parse, endpoint="execute", args={ "lua_source": login_script, # real credentials belong in environment variables, not source "username": "user123", "password": "password", }, ) def parse(self, response): if not response.data.get("logged_in"): self.logger.warning("login did not complete in time") return confirmed = "Logged in as User123" in response.data["html"] yield {"logged_in": confirmed} ``` The public sample credentials are `user123` and `password`. Against them, the spider yields `{'logged_in': True}`. The `logged_in` flag comes from Splash polling for the `#secret-message` element in a bounded loop, so the callback only trusts a login that finished the `Logged in as User123` state instead of assuming `submit()` alone was enough. The reusable pattern passes credentials through request arguments. Real credentials belong in environment variables, and they should stay out of `lua_source` string interpolation, logs, and screenshots. ## What happened when Splash 3.5 scrolled the testimonials page? In a bounded test the public testimonials target stayed at its initial 10 records after five scroll-and-wait cycles. This run did not isolate whether timing, script behavior, target behavior, or the renderer caused the unchanged collection: lua```lua function main(splash, args) splash:go(args.url) splash:wait(1.0) local scroll = splash:jsfunc([[ function scrollToBottom() { window.scrollTo(0, document.body.scrollHeight); } ]]) for i = 1, args.scrolls do scroll() splash:wait(1.0) end return { html = splash:html() } end ``` Run through `SplashRequest` with `args.scrolls` set to 5, this script still returns 10 testimonial elements. The same count as the first render before any scrolling happened. ## What breaks in Scrapy Splash and how do you debug it? | Symptom | Boundary | Next action | |---|---|---| | Import error for `canonicalize_url` | Incompatible Scrapy version (2.16.0 and later) | Pin Scrapy to 2.12.0, or track scrapy-splash issue 332 for a fix | | `AttributeError` or deprecation warning on `_get_slot_key` | Newer Scrapy downloader internals. This warning fires on the pinned stack today | Treat it as an early signal, not a current failure, and re-test before upgrading Scrapy | | Startup middleware error | An async-only spider paired with the older Splash spider middleware | Check spider middleware order against the settings block above | | Lua 400 response | Lua syntax error or a missing argument | Re-check `splash.args` names against what the Lua script reads | | `nil` element from `splash:select` | Selector did not match the rendered DOM | Inspect the returned HTML directly before trusting the selector | | 200 status but missing dynamic records | The tested page state did not load on this stack | Inspect target requests and wait times before assigning a cause | | Container does not start on another architecture | Host platform does not match the pulled image | Compare the host platform with the image tag and validate a suitable image separately | Don't fix any of these by editing package source files. Every row above resolves through a version pin, a settings change or more diagnostic wait time. ## When should you move from Splash to Playwright or managed rendering? ### When is Scrapy Playwright the better browser route? Start a new project on Scrapy Playwright instead of Splash. Playwright runs a current Chromium engine with modern waits, locators, and browser contexts, which covers the complex interaction Splash's older engine can't match. [Scrapy Playwright Tutorial: Scrape Dynamic WebsitesLearn how to use scrapy-playwright with Scrapy 2.18 to scrape JavaScript pages, wait for dynamic content, manage browser contexts, and run on Windows.](https://scrapfly.io/blog/posts/how-to-use-scrapy-with-playwright) ### When should a Splash job use a standalone Scrapfly product? For managed HTTP fetching and JavaScript rendering where the output is a rendered response, [Web Scraping API](https://scrapfly.io/products/web-scraping-api) replaces the Splash server entirely, with no container to run and no version to pin. For click, form, and session workflows that need a current managed browser, [Cloud Browser](https://scrapfly.io/products/cloud-browser-api) covers that with no Scrapy integration to maintain. Splash still has a place for maintaining what already runs on it. Neither of these paths asks you to keep patching an aging rendering server. ### Power your scraping with Scrapfly Forget about getting blocked. Scrapfly handles anti-bot bypasses, browser rendering, and proxy rotation so you can focus on the data. [Try for FREE!](https://scrapfly.io/register) ## FAQ What is the difference between Scrapy Splash and headless browsers?Scrapy Splash routes requests through a separate rendering server. A headless browser like [Selenium](https://scrapfly.io/blog/posts/web-scraping-with-selenium-and-python) or [Puppeteer](https://scrapfly.io/blog/posts/web-scraping-with-puppeteer-and-nodejs) embeds the engine in your scraping process instead, the same way Playwright does. Can I still scrape JavaScript-rendered pages with Scrapy using Splash?Yes, on the pinned stack in this guide. This guide confirms `scrapy-splash` 0.11.1 paired with Scrapy 2.12.0 renders JavaScript through Splash. Does Scrapy Splash bypass anti-bot protection?No. Splash ships no bot-detection bypass, so a target with [anti-bot protection](https://scrapfly.io/blog/posts/how-to-bypass-anti-bot-protection-when-web-scraping) blocks a Splash request the same as a plain one. Can Splash return a PNG screenshot to Scrapy?Yes. The `render.png` endpoint returns PNG output and accepts the same render arguments as `render.html`, separate from the HTML parsing examples above. Can scrapy-splash preserve cookies between requests?Yes, within this pinned stack. `SplashCookiesMiddleware` handles cookie transfer between Splash requests, though this guide doesn't extend that into full session management. Is web scraping with Scrapy Splash legal?Scraping publicly available data is generally legal in many jurisdictions, but rules vary by site, region, and how you use the data. Check a target's terms of service and applicable law before scraping it. ## Summary Scrapy Splash still runs on a frozen stack: Python 3.12, Scrapy 2.12.0, Twisted 24.11.0, `scrapy-splash` 0.11.1, and Splash 3.5. The render request and the form login both work on it, verified live against the public test pages in this guide. `scrapy-splash` 0.11.1 breaks the moment you move to Scrapy 2.16.0 or later, including the current 2.18.0 release. The bounded scroll test stayed at 10 records. A limit this guide states instead of hiding. If you are maintaining an existing splash crawler. Keep the version pins above and treat any scrapy or twisted upgrade as a reason to re-test. For a new browser automation project Start with playwright or move the rendering to a managed web scraping API instead. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [Does Scrapy Splash still work in 2026?](#does-scrapy-splash-still-work-in-2026) - [How do you run Splash 3.5 with Docker?](#how-do-you-run-splash-3-5-with-docker) - [How do you start the pinned Splash image?](#how-do-you-start-the-pinned-splash-image) - [What does the Splash 3.5 runtime limit?](#what-does-the-splash-3-5-runtime-limit) - [How do you install the pinned scrapy-splash stack?](#how-do-you-install-the-pinned-scrapy-splash-stack) - [Which Python packages must stay pinned together?](#which-python-packages-must-stay-pinned-together) - [Why does scrapy-splash fail on Scrapy 2.16.0 and later?](#why-does-scrapy-splash-fail-on-scrapy-2-16-0-and-later) - [How do you configure scrapy-splash for the pinned Scrapy project?](#how-do-you-configure-scrapy-splash-for-the-pinned-scrapy-project) - [Which Splash middlewares and fingerprinter should settings.py use?](#which-splash-middlewares-and-fingerprinter-should-settings-py-use) - [How do you render one page with SplashRequest?](#how-do-you-render-one-page-with-splashrequest) - [How does a Scrapy request reach the Splash HTTP API?](#how-does-a-scrapy-request-reach-the-splash-http-api) - [What is the difference between the request to Splash and the request to the target?](#what-is-the-difference-between-the-request-to-splash-and-the-request-to-the-target) - [How do you run Lua scripts through scrapy-splash?](#how-do-you-run-lua-scripts-through-scrapy-splash) - [How does SplashRequest call the execute endpoint?](#how-does-splashrequest-call-the-execute-endpoint) - [How do you forward a target POST from a Splash Lua script?](#how-do-you-forward-a-target-post-from-a-splash-lua-script) - [How do you submit a login form with Scrapy Splash?](#how-do-you-submit-a-login-form-with-scrapy-splash) - [How do Splash form fill and submit work together?](#how-do-splash-form-fill-and-submit-work-together) - [What happened when Splash 3.5 scrolled the testimonials page?](#what-happened-when-splash-3-5-scrolled-the-testimonials-page) - [What breaks in Scrapy Splash and how do you debug it?](#what-breaks-in-scrapy-splash-and-how-do-you-debug-it) - [When should you move from Splash to Playwright or managed rendering?](#when-should-you-move-from-splash-to-playwright-or-managed-rendering) - [When is Scrapy Playwright the better browser route?](#when-is-scrapy-playwright-the-better-browser-route) - [When should a Splash job use a standalone Scrapfly product?](#when-should-a-splash-job-use-a-standalone-scrapfly-product) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ blocking proxies ### TCP/IP Stack Fingerprinting and Proxy Bypass How passive TCP/IP fingerprinting exposes OS mismatches, proxy leaks, and VPN overhead before HTTP parsing starts, and h... ](https://scrapfly.io/blog/posts/tcp-ip-stack-fingerprinting-proxy-bypass) [ ### How Browser Fingerprinting Works and How to Defend Against It Learn how browser fingerprinting works, from canvas to WebGPU, test your fingerprint with free tools, and apply develope... ](https://scrapfly.io/blog/posts/how-browser-fingerprinting-works) [ python xpath ### Web Scraping With Scrapy: The Complete Guide in 2026 Build and run a Scrapy 2.18 project with async start(), pagination, selector tests, item validation, pipelines, and JSON... ](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) ## Related Questions - [ Q Scrapy vs BeautifulSoup: Which Should You Use? ](https://scrapfly.io/blog/answers/scrapy-vs-beautifulsoup) - [ Q How to find elements by CSS selectors in Playwright? ](https://scrapfly.io/blog/answers/how-to-find-elements-by-css-selectors-in-playwright) - [ Q How to find elements by XPath in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-find-elements-by-xpath-in-puppeteer) - [ Q How to add headers to every or some scrapy requests? ](https://scrapfly.io/blog/answers/how-to-add-headers-to-every-or-some-scrapy-requests) Run headless browsers at scale, **1,000 free credits** [Start Free](https://scrapfly.io/register)