[Blog](https://scrapfly.io/blog) / [headless-browser](https://scrapfly.io/blog/tag/headless-browser) / [Scrapy Playwright Tutorial: Scrape Dynamic Websites](https://scrapfly.io/blog/posts/how-to-use-scrapy-with-playwright) # Scrapy Playwright Tutorial: Scrape Dynamic Websites by [Mazen Ramadan](https://scrapfly.io/blog/author/mazen) Aug 27, 2026 19 min read [\#headless-browser](https://scrapfly.io/blog/tag/headless-browser) [\#playwright](https://scrapfly.io/blog/tag/playwright) [\#python](https://scrapfly.io/blog/tag/python) [\#scrapy](https://scrapfly.io/blog/tag/scrapy) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-use-scrapy-with-playwright "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-use-scrapy-with-playwright&text=Scrapy%20Playwright%20Tutorial%3A%20Scrape%20Dynamic%20Websites "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-use-scrapy-with-playwright "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-use-scrapy-with-playwright) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-use-scrapy-with-playwright) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-use-scrapy-with-playwright) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-use-scrapy-with-playwright) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-use-scrapy-with-playwright) Most Scrapy requests should not open a browser. On the testimonials fixture, the initial HTML already contained 10 records and exposed a paginated endpoint that returned all 60 without Playwright. Use scrapy-playwright only when the data appears after browser interaction or you cannot reproduce the background request. The examples below run on Scrapy 2.18.0, scrapy-playwright 0.0.48, and Playwright 1.62.0. Let's dive in! ## Key Takeaways - scrapy-playwright is a Scrapy download handler, not downloader middleware. Only requests with `meta={"playwright": True}` open a browser page. - Scrapy 2.18 uses `async def start()`. Support for a spider-defined `start_requests()` path was removed in Scrapy 2.16. - Inspect the HTML and background requests first. A direct Scrapy request is cheaper when it can reproduce the data source. - Prefer callable `PageMethod` actions with Playwright Locators and application-specific conditions over fixed sleeps or `networkidle` as a readiness signal. - `playwright_include_page=True` transfers page ownership to the callback. Close the page in both the callback and errback or the crawl can exhaust its page limit. - Current scrapy-playwright supports native Windows through a separate ProactorEventLoop thread. WSL is optional, not required. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## When Should a Scrapy Request Use Playwright? Use the normal Scrapy downloader whenever the initial HTML or a reproducible background request already contains the data. Add `meta={"playwright": True}` only to requests that need browser rendering or interaction. | What you observe | Best Scrapy route | Why | |---|---|---| | Data is present in the response HTML | Normal Scrapy request | No browser process is needed | | The page calls a JSON or HTML endpoint you can reproduce | Request that endpoint with Scrapy | Lower CPU, memory, and bandwidth per page | | Content appears only after JavaScript runs | scrapy-playwright request | The browser builds the rendered DOM | | The workflow requires clicking, filling, scrolling, or browser state | scrapy-playwright with PageMethods or a Page callback | The browser performs the interaction | [web-scraping.dev](https://web-scraping.dev) is a public sandbox site built for practicing scraping techniques and its testimonials page makes a good concrete check: Reviews on web-scraping.dev- `https://web-scraping.dev/testimonials` returns 10 testimonial cards in the initial HTML. - Its last card exposes an `hx-get` endpoint. Replaying the five subsequent pages with the fixture's Referer and public test token returns 60 records total. - A browser can also load the same 60 records, but it is not the efficient route when the endpoint is reproducible. This finding belongs to this test fixture. Not every dynamic site exposes an easy endpoint to replay, so check each target on its own terms before assuming one approach covers it. For more on locating a reproducible data source before reaching for a browser, see the [Web Scraping With Scrapy: The Complete Guide in 2026Build and run a Scrapy 2.18 project with async start(), pagination, selector tests, item validation, pipelines, and JSON Lines export.](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) Once you have confirmed a request actually needs a browser, the next step is getting scrapy-playwright installed and wired into the project. ## How Do You Install scrapy-playwright for Scrapy 2.18? These examples were built against one declared version matrix: - Python 3.14.7 - Scrapy 2.18.0 - scrapy-playwright 0.0.48 - Playwright Python 1.62.0 - Playwright's managed Chromium-family browser bundle Install the three Python packages, then install the browser binary: shell```shell pip install scrapy scrapy-playwright playwright playwright install chromium ``` scrapy-playwright already declares Playwright as a dependency. The explicit Playwright pin above just keeps this tutorial reproducible. `playwright install chromium` installs Playwright's own managed Chromium-family bundle, not the system Chrome stable channel, and the exact headed or headless binary varies by platform across Playwright releases. Linux and CI images often need extra system packages alongside the browser binary. Use `playwright install --with-deps chromium` there instead, as documented in [Playwright's browser installation guide](https://playwright.dev/python/docs/browsers). Treat it as the Linux and CI variant not a universal replacement for the Windows and macOS command above. One dated note worth keeping in mind: Playwright 1.62 no longer supports Debian 11. With the packages installed, start a project: Create a new Scrapy project through the `Scrapy` commands: shell```shell $ scrapy startproject reviewgather reviewgather-scraper # ^ name ^ project directory ``` Then, wire the download handler into `settings.py` python```python # settings.py DOWNLOAD_HANDLERS = { "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", } TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor" PLAYWRIGHT_BROWSER_TYPE = "chromium" PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = 30_000 PLAYWRIGHT_MAX_PAGES_PER_CONTEXT = 4 USER_AGENT = None ``` Each setting does one job: - `ScrapyPlaywrightDownloadHandler` is a download handler not middleware. Requests without `meta["playwright"]` still use Scrapy's normal HTTP handler through inheritance. - Registering only `"https"` is enough for the examples in this article and avoids standing up a second, independent handler instance for plain HTTP. - `TWISTED_REACTOR` has been the default in new Scrapy projects since 2.7, but declaring it explicitly here keeps the integration requirement visible. - `USER_AGENT = None` lets the browser present its own native User-Agent instead of Scrapy's default identity string. - `PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT` controls page navigation. `PLAYWRIGHT_LAUNCH_OPTIONS["timeout"]` is the separate setting for browser launch, and it is not used in these examples. Nowhere in this article is scrapy-playwright called middleware. It is a download handler, and that distinction is what the next section builds on. With the handler registered, the next step is sending one request through it and confirming the browser actually rendered something the plain HTTP handler could not. ## How do you render JavaScript with Scrapy Playwright? Standard HTTP clients receive only the initial unrendered HTML shell, Playwright allows the JavaScript execution engine to build the dynamic DOM elements before extraction takes place. python```python import scrapy from scrapy_playwright.page import PageMethod async def wait_for_quotes(page): await page.locator("div.quote").first.wait_for(state="visible") class QuotesSpider(scrapy.Spider): name = "quotes" async def start(self): yield scrapy.Request( url="https://quotes.toscrape.com/js/", meta={ "playwright": True, "playwright_page_methods": [ PageMethod(wait_for_quotes), ], }, ) def parse(self, response): for quote in response.css("div.quote")[:3]: text = quote.css("span.text::text").get() text = text.replace("“", "").replace("”", "").strip() if text else "" author = quote.css("small.author::text").get() tags = quote.css("div.tags a.tag::text").getall() yield { "text": text, "author": author, "tags": tags, } ``` Execute this spider and save the output directly using: bash```bash scrapy crawl quotes -O quotes.json ``` Example output: json```json [ {"text": "The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.", "author": "Albert Einstein", "tags": ["change", "deep-thoughts", "thinking", "world"]}, {"text": "It is our choices, Harry, that show what we truly are, far more than our abilities.", "author": "J.K. Rowling", "tags": ["abilities", "choices"]}, {"text": "There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.", "author": "Albert Einstein", "tags": ["inspirational", "life", "live", "miracle", "miracles"]} ] ``` Execution mechanism - `meta["playwright"]` routes this request through the Playwright-backed download handler. - The callable PageMethod receives a Playwright Page and waits on a Locator condition. - scrapy-playwright builds the final Scrapy response after the PageMethods finish, so normal Scrapy CSS/XPath extraction still works. - No Page object escapes into the callback, so scrapy-playwright closes it automatically. Once the browser hands back a fully rendered response, extraction is normal Scrapy CSS selection, covered in more depth in the [CSS selector guide](https://scrapfly.io/blog/posts/parsing-html-with-css). PageMethods are what get a request to that point whenever a single wait is not enough ## How do PageMethods work in scrapy-playwright? PageMethods run in list order before the Scrapy callback receives its response. A PageMethod may name a Playwright Page method or receive a callable whose first argument is the Page. | Request metadata | Purpose | Page ownership | |---|---|---| | `playwright=True` | Render this request with Playwright | scrapy-playwright closes the page | | `playwright_page_methods=[...]` | Run waits, clicks, fills, screenshots, or callables before the callback | scrapy-playwright closes the page unless included | | `playwright_context="name"` | Choose a browser context | Context remains until explicitly closed or the handler shuts down | | `playwright_context_kwargs={...}` | Configure a new named context | Ignored if that context already exists | | `playwright_include_page=True` | Expose the Page in the callback | Your callback and errback must close it | Two details matter once you start reading these results back. - Each result is available on the corresponding object in `response.meta["playwright_page_methods"]`, in the same order the PageMethods were passed in. - If a PageMethod navigates, `response.url` can reflect the last navigation instead of the original request URL, and other response attributes can shift along with it. The two examples below put a callable PageMethod to work through Playwright Locators, first on a login form and then on a scroll-triggered testimonials list. ### How do you click and fill forms with Playwright Locators? Prefer one callable PageMethod that owns an entire interaction over a chain of separate click and fill PageMethod entries. A callable can wait between steps with a Locator condition and hand a result back to the callback, which a flat list of PageMethod calls cannot do on its own. python```python import scrapy from scrapy_playwright.page import PageMethod async def complete_login(page): cookie_button = page.locator("button#cookie-ok") if await cookie_button.count(): await cookie_button.click() await page.locator("input[name='username']").fill("user123") await page.locator("input[name='password']").fill("password") await page.locator("button[type='submit']").click() await page.locator("#secret-message").wait_for(state="visible") return page.url class LoginSpider(scrapy.Spider): name = "login" custom_settings = { "PLAYWRIGHT_PROCESS_REQUEST_HEADERS": None, } async def start(self): yield scrapy.Request( url="https://web-scraping.dev/login?cookies=", meta={ "playwright": True, "playwright_page_methods": [ PageMethod(complete_login), ], }, ) def parse(self, response): login_step = response.meta["playwright_page_methods"][0] yield { "logged_in": login_step.result is not None, "url": login_step.result, } ``` Run it the same way as the quotes spider: bash```bash scrapy crawl login -O login.json ``` Observed output: json```json [ {"logged_in": true, "url": "https://web-scraping.dev/login?cookies="} ] ``` `PLAYWRIGHT_PROCESS_REQUEST_HEADERS = None` is scoped to this spider's `custom_settings`, not the shared `settings.py` baseline. Without it Scrapy's headers override the browser-set cookie this login depends on and the request times out instead of logging in. Setting it to `None` hands Playwright full control of headers at the cost of ignoring `Request.headers` and `Request.cookies`. Treat it as a fixture-specific override, not a global default. ### How do you scroll with scrapy-playwright until dynamic content is complete? The testimonials fixture loads more cards through an htmx `hx-trigger="revealed"` request tied to the last `div.testimonial` card, so scrolling that last card into view is what triggers each additional page. A condition-based loop that waits for the DOM count to increase is the reliable way to drive that, rather than a fixed number of scrolls with a sleep between them. python```python import scrapy from playwright.async_api import TimeoutError as PlaywrightTimeoutError from scrapy_playwright.page import PageMethod async def load_all_testimonials(page, expected): cards = page.locator("div.testimonial") for _ in range(8): count = await cards.count() if count >= expected: break await cards.last.scroll_into_view_if_needed() try: await page.wait_for_function( "(n) => document.querySelectorAll('div.testimonial').length > n", arg=count, timeout=5000, ) except PlaywrightTimeoutError: break final_count = await cards.count() if final_count < expected: raise RuntimeError(f"expected {expected} testimonials, loaded {final_count}") class TestimonialsSpider(scrapy.Spider): name = "testimonials" async def start(self): yield scrapy.Request( url="https://web-scraping.dev/testimonials", meta={ "playwright": True, "playwright_page_methods": [ PageMethod(load_all_testimonials, 60), PageMethod("screenshot", path="testimonials.png", full_page=True), ], }, ) def parse(self, response): cards = response.css("div.testimonial") yield { "count": len(cards), "first_three": [ c.css("p.text::text").get(default="").strip() for c in cards[:3] ], } ``` The screenshot PageMethod runs last in the list, so it fires only after `load_all_testimonials` confirms the DOM already holds 60 cards, not after a fixed delay. On this fixture, the rendered response contained all 60 testimonial cards, and the full-page screenshot was written only after the 60th card had loaded. The first three records matched the 10-card initial non-browser HTTPS response exactly, which confirms those first cards never needed a browser in the first place. The `60` here belongs to this fixture, not to scrolling in general. On a real target, replace it with a condition that comes from the application itself, such as an end-of-list marker, a disabled "next" control, a known API exhaustion signal, or a count read from the page rather than hardcoded. ## How should scrapy-playwright wait for dynamic content? Prefer Playwright's auto-waiting plus a Locator, URL, response, or application-state condition that proves the data you need is actually ready, over any form of fixed delay. | Wait | Use | Avoid | |---|---|---| | Locator action or `locator.wait_for()` | Element must be actionable or visible | Generic selector sleeps | | `page.wait_for_url()` | A navigation has a known destination | Guessing a delay after a click | | `page.expect_response()` or application state | A known request or result marks completion | Treating page load as data readiness | | `page.wait_for_function()` | A target-specific DOM or state condition | Open-ended polling | | `page.wait_for_timeout()` | Debugging only | Production use, wait for a locator, network event, URL, or application condition instead | | `networkidle` | Rare target-specific cases with a known quiet network | General dynamic-app readiness | These are condition-based waits, not dynamic timeouts. PageMethod ordering matters here too. PageMethods run after scrapy-playwright's own initial navigation has already reached `domcontentloaded`, so adding a second `domcontentloaded` wait as the first PageMethod is usually redundant. ## How do headers, cookies, and browser contexts work in scrapy-playwright? - The default header processor takes Scrapy request data and applies Scrapy headers to navigation requests, while subresource requests generally keep browser headers with the Scrapy User-Agent applied on top. - `USER_AGENT = None` in `settings.py` lets the browser use its own native User-Agent instead of Scrapy's default identity string. - `PLAYWRIGHT_PROCESS_REQUEST_HEADERS = None`, as used above, gives Playwright full control, which means `Request.headers`, Scrapy-added headers, and `Request.cookies` are all ignored. - Browser-context cookies belong to the Playwright context, not to Scrapy. Define context state with `PLAYWRIGHT_CONTEXTS` or a request's `playwright_context_kwargs` when a workflow needs it. - Named contexts are reused across requests. New context kwargs passed on a later request are ignored once that named context already exists. A custom header processor must use scrapy-playwright's current keyword-only signature: python```python async def add_project_header(*, browser_type_name, playwright_request, scrapy_request_data): headers = await playwright_request.all_headers() headers["x-project"] = "scrapy-playwright-tutorial" return headers ``` Point `PLAYWRIGHT_PROCESS_REQUEST_HEADERS` at this callable's import path. It runs for every request the handler processes not only the first navigation. The old positional signature, `browser_type`, `playwright_request`, `scrapy_headers`, fails outright on scrapy-playwright 0.0.48 because the handler now calls it with `browser_type_name` as a keyword argument. Browser contexts carry locale, viewport, and geolocation state the same way. The [language, currency, and location guide](https://scrapfly.io/blog/posts/how-to-scrape-in-another-language-or-currency) covers configuring that context state for a target that responds differently per region. Scrapfly #### Need a cloud browser for scraping? Run headless browsers at scale with Scrapfly Cloud Browser — no infrastructure to manage. [Try Free →](https://scrapfly.io/register)## How do you close Playwright pages without stalling Scrapy? Do not request `playwright_include_page=True` unless the callback truly needs to call the Page object directly. PageMethods already cover clicks, fills, waits, and screenshots without it, and scrapy-playwright closes the page automatically once they finish. python```python import scrapy class LifecycleSpider(scrapy.Spider): name = "lifecycle" custom_settings = {"PLAYWRIGHT_MAX_PAGES_PER_CONTEXT": 1} async def start(self): yield scrapy.Request( url="https://quotes.toscrape.com/js/", meta={"playwright": True, "playwright_include_page": True}, callback=self.parse, errback=self.errback, ) async def parse(self, response): page = response.meta["playwright_page"] try: title = await page.title() yield {"title": title} finally: await page.close() async def errback(self, failure): page = failure.request.meta.get("playwright_page") if page is not None and not page.is_closed(): await page.close() ``` The callback wraps its direct Page use in `try/finally` so the page closes whether extraction succeeds or raises. The errback retrieves the Page defensively, since a failure can happen before `playwright_page` ever lands in `response.meta`, and closes it only when it is both present and still open. `PLAYWRIGHT_MAX_PAGES_PER_CONTEXT = 1` is deliberate here. Open pages count toward that limit, so a callback that forgets to close its page stalls later requests instead of failing loudly, which is why both the callback and errback above close it. Close a page before its context and keep retries on Scrapy's normal handling rather than a separate Playwright-level loop. ## Does scrapy-playwright work on Windows in 2026? Yes. Current scrapy-playwright supports native Windows directly. It runs Playwright's subprocess-capable `ProactorEventLoop` on a separate thread, while Scrapy and Twisted keep using `SelectorEventLoop` for the asyncio reactor on the main thread. WSL remains an optional environment for this integration, not a requirement. One qualifier is worth knowing here. The optional scrapy-playwright memory-usage extension is not available on Windows, because it depends on Python's `resource` module, which Windows does not provide. See the [upstream Windows support notes](https://github.com/scrapy-plugins/scrapy-playwright/blob/v0.0.48/README.md#windows-support) for the current state of that gap. ## How do you keep scrapy-playwright efficient in a larger crawl? 1. Set `PLAYWRIGHT_MAX_PAGES_PER_CONTEXT` deliberately for the workload, and add `PLAYWRIGHT_MAX_CONTEXTS` once several named contexts can exist at the same time. 2. Block only the resources a target genuinely does not need through `PLAYWRIGHT_ABORT_REQUEST`, and track the `playwright/request_count/aborted` stat rather than assuming the rule ran the way you expect. 3. Reuse browser contexts where state should persist across requests, but close contexts you created dynamically once their job is done. 4. Monitor the browser subprocess on its own terms. Scrapy's default memory extension does not account for Playwright's browser process, and the optional scrapy-playwright replacement only adds that on supported platforms. 5. Record scrapy-playwright's own request, response, page, and context stats alongside the crawl output you already use for debugging. None of this changes the basic economics of a browser request. scrapy-playwright does not preserve Scrapy's normal HTTP throughput, and a browser-rendered request stays materially heavier than a plain one, so the decision table at the top of this article is worth rechecking before every new `meta={"playwright": True}` you add. ## When should you use standalone Playwright or Scrapfly Cloud Browser? Three routes cover most crawls: 1. **Normal Scrapy** when the data is already in the HTML or behind a reproducible endpoint. 2. **scrapy-playwright** when a crawl benefits from Scrapy's scheduling and pipelines, and only some requests need rendering or interaction, exactly like the examples above. 3. **Direct Playwright with a managed browser** when the job is primarily one stateful browser workflow, or when operating browsers, proxies, sessions, and bot-protection handling has itself become the infrastructure problem. For browser-only jobs without Scrapy's scheduler, see the [standalone Playwright Python guide](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python). Scrapfly Cloud Browser provides remote Chromium sessions over a CDP WebSocket for direct Playwright clients. Use it when browser-fleet, proxy, session, or anti-bot infrastructure is the problem rather than Scrapy scheduling. The example below connects to Cloud Browser directly through Playwright's CDP client inside a Scrapy spider. It does not go through scrapy-playwright's download handler at all. python```python import asyncio import os import platform import threading from concurrent.futures import Future from urllib.parse import urlencode import scrapy from playwright.async_api import async_playwright async def scrape_products(endpoint): browser = context = page = None async with async_playwright() as playwright: try: browser = await playwright.chromium.connect_over_cdp(endpoint) if not browser.contexts: raise RuntimeError("Cloud Browser exposed no default context") context = browser.contexts[0] page = context.pages[0] if context.pages else await context.new_page() await page.goto( "https://web-scraping.dev/products", wait_until="domcontentloaded", ) products = page.locator(".product") await products.first.wait_for(state="visible") first = products.first return { "title": (await first.locator("h3 a").inner_text()).strip(), "price": (await first.locator(".price").inner_text()).strip(), "product_count": await products.count(), "source": page.url, } finally: if page is not None and not page.is_closed(): await page.close() if context is not None: await context.close() if browser is not None and browser.is_connected(): await browser.close() def run_in_proactor_thread(coro, result: Future): loop = asyncio.WindowsProactorEventLoopPolicy().new_event_loop() try: result.set_result(loop.run_until_complete(coro)) except Exception as exc: result.set_exception(exc) finally: loop.close() class RemoteBrowserDirectSpider(scrapy.Spider): name = "remote_browser_direct" custom_settings = { "TWISTED_REACTOR": ( "twisted.internet.asyncioreactor.AsyncioSelectorReactor" ), } async def start(self): endpoint = "wss://browser.scrapfly.io?" + urlencode( { "api_key": os.environ["SCRAPFLY_API_KEY"], "proxy_pool": "public_datacenter_pool", "os": "linux", "browser_brand": "chrome", "timeout": "60", } ) coro = scrape_products(endpoint) if platform.system() == "Windows": result: Future = Future() threading.Thread( target=run_in_proactor_thread, args=(coro, result), daemon=True ).start() item = await asyncio.wrap_future(result) else: item = await coro yield item ``` Run it with `SCRAPFLY_API_KEY` set in the environment. On Python 3.14.7, Scrapy 2.18.0, and Playwright 1.62.0, this example yielded the following item and closed the page, default context, and browser connection normally: json```json { "title": "Box of Chocolate Candy", "price": "24.99", "product_count": 5, "source": "https://web-scraping.dev/products" } ``` The `platform.system() == "Windows"` branch is not optional. Playwright's driver launches a local subprocess even for `connect_over_cdp` and Windows' `SelectorEventLoop`, which `AsyncioSelectorReactor` forces Scrapy onto cannot launch subprocesses. The fix mirrors scrapy-playwright's own workaround: - run Playwright on a dedicated thread with a `ProactorEventLoop`. - bridge the result back with `asyncio.wrap_future`. Linux and macOS have no such restriction, so the coroutine runs directly there. ## Conclusion This is a direct Playwright CDP client running inside a Scrapy spider, not out-of-the-box scrapy-playwright download-handler support for Scrapfly Cloud Browser. Using a browser here managed or not, is not itself a guarantee against anti-bot defenses. Between the three routes, the decision comes down to what Scrapy is actually giving you. If a crawl needs Scrapy's scheduler, retries, and pipelines with only a handful of requests needing a browser, scrapy-playwright is the direct fit demonstrated throughout this article. If the job is one continuous browser workflow, or the infrastructure around browsers has become its own project, reach for [Cloud Browser API](https://scrapfly.io/products/cloud-browser-api) and its [CDP getting-started documentation](https://scrapfly.io/docs/cloud-browser-api/getting-started) instead. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [When Should a Scrapy Request Use Playwright?](#when-should-a-scrapy-request-use-playwright) - [How Do You Install scrapy-playwright for Scrapy 2.18?](#how-do-you-install-scrapy-playwright-for-scrapy-2-18) - [How do you render JavaScript with Scrapy Playwright?](#how-do-you-render-javascript-with-scrapy-playwright) - [How do PageMethods work in scrapy-playwright?](#how-do-pagemethods-work-in-scrapy-playwright) - [How do you click and fill forms with Playwright Locators?](#how-do-you-click-and-fill-forms-with-playwright-locators) - [How do you scroll with scrapy-playwright until dynamic content is complete?](#how-do-you-scroll-with-scrapy-playwright-until-dynamic-content-is-complete) - [How should scrapy-playwright wait for dynamic content?](#how-should-scrapy-playwright-wait-for-dynamic-content) - [How do headers, cookies, and browser contexts work in scrapy-playwright?](#how-do-headers-cookies-and-browser-contexts-work-in-scrapy-playwright) - [How do you close Playwright pages without stalling Scrapy?](#how-do-you-close-playwright-pages-without-stalling-scrapy) - [Does scrapy-playwright work on Windows in 2026?](#does-scrapy-playwright-work-on-windows-in-2026) - [How do you keep scrapy-playwright efficient in a larger crawl?](#how-do-you-keep-scrapy-playwright-efficient-in-a-larger-crawl) - [When should you use standalone Playwright or Scrapfly Cloud Browser?](#when-should-you-use-standalone-playwright-or-scrapfly-cloud-browser) - [Conclusion](#conclusion) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ python headless-browser ### How to Scrape Dynamic Websites Using Headless Web Browsers Introduction to using web automation tools such as Puppeteer, Playwright, Selenium and ScrapFly to render dynamic websit... ](https://scrapfly.io/blog/posts/scraping-using-browsers) [ python xpath ### Web Scraping With Scrapy: The Complete Guide in 2026 Build and run a Scrapy 2.18 project with async start(), pagination, selector tests, item validation, pipelines, and JSON... ](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) [ blocking ### What is CreepJS Browser Fingerprint and How to Bypass It In this article, we will explore the inner workings of CreepJS, one of the prominent browser fingerprinting tools and ho... ](https://scrapfly.io/blog/posts/browser-fingerprinting-with-creepjs) ## Related Questions - [ Q How to wait for page to load in Playwright? ](https://scrapfly.io/blog/answers/how-to-wait-for-page-to-load-in-playwright) - [ Q How to wait for a page to load in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-wait-for-page-to-load-in-puppeteer) - [ Q How to wait for page to load in Selenium? ](https://scrapfly.io/blog/answers/how-to-wait-for-page-to-load-in-selenium) - [ Q How to get page source in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-get-page-source-in-puppeteer) Run headless browsers at scale, **1,000 free credits** [Start Free](https://scrapfly.io/register)