[Blog](https://scrapfly.io/blog) / [headless-browser](https://scrapfly.io/blog/tag/headless-browser) / [Web Scraping With Playwright in 2026: A Python Guide](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python) # Web Scraping With Playwright in 2026: A Python Guide by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) Sep 03, 2026 29 min read [\#headless-browser](https://scrapfly.io/blog/tag/headless-browser) [\#python](https://scrapfly.io/blog/tag/python) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-playwright-and-python "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-playwright-and-python&text=Web%20Scraping%20With%20Playwright%20in%202026%3A%20A%20Python%20Guide "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-playwright-and-python "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-playwright-and-python) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-playwright-and-python) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-playwright-and-python) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-playwright-and-python) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-playwright-and-python) You call `requests.get()` on a listing page and get 200 back. Your selectors match nothing, because the data arrives after JavaScript runs, and `requests` never executes it. Playwright drives a real browser, so the content is there before you read it. By the end you'll have a scraper that waits for content, extracts it with locators or a parser, and fills out forms. It also scrolls, paginates, and rewrites its own requests. You'll see where running your own browser stops paying for itself. [How to Scrape Dynamic Websites Using Headless Web BrowsersIntroduction to using web automation tools such as Puppeteer, Playwright, Selenium and ScrapFly to render dynamic websites for web scraping](https://scrapfly.io/blog/posts/scraping-using-browsers) ## Key Takeaways - **Wait for a selector, not a timer.** Reach for `wait_for_timeout()` only when debugging - **`locator.all()` doesn't wait.** It returns whatever matches right now - **Only `page.route()` changes a request.** `page.on()` observes traffic and can't modify it - **Browser-managed headers resist rewriting.** Referer stays the browser's own value - **Block images, fonts, and media by default.** Scripts are riskier to block - **Check for a background API first.** Skipping the browser is often fastest - **`connect_over_cdp()` moves the browser process off your machine.** Locators and waits stay familiar, but CDP has lower fidelity than Playwright's native protocol and the Cloud Browser limitations below still apply **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## What is Playwright? Playwright is a browser automation library that drives Chromium, Firefox, and WebKit through one API. For Chromium it speaks the Chrome DevTools Protocol (CDP). Firefox and WebKit run patched builds with their own low-level protocols instead. None of the three go through the WebDriver protocol that Selenium uses. It ships both a synchronous and an asynchronous Python client. Prototype with the sync client, then move to the async one when you need to scale. A real browser saves you from reverse engineering a site's private API. It runs the page's JavaScript exactly as a visitor's browser would, so content that only exists after a script runs still shows up. That's also the tradeoff. A browser costs far more time and memory per page than a plain HTTP request, so reach for Playwright only when a page needs it. ### Playwright vs Selenium vs Puppeteer Playwright covers more languages than [Puppeteer](https://pptr.dev/), JavaScript and TypeScript only, and talks to the browser over CDP, where [Selenium](https://www.selenium.dev/) uses WebDriver. Playwright ships sync and async clients. Selenium is sync only, Puppeteer async only. For a closer look at either alternative, see our guides on [Playwright vs Selenium](https://scrapfly.io/blog/posts/playwright-vs-selenium) and [Puppeteer vs Playwright](https://scrapfly.io/blog/posts/puppeteer-vs-playwright). Writing in JavaScript instead of Python? See [Playwright with JavaScript](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-javascript). With the comparison out of the way, here's how to get Playwright itself installed and running. ## Setup This guide targets **Playwright 1.62.0**, released July 31, 2026, and needs Python 3.10 or newer. Check your version before you install if you're on an older Python. Install the package and the Chromium browser it drives: bash```bash pip install playwright parsel beautifulsoup4 playwright install chromium ``` `playwright install firefox` and `playwright install webkit` add the other two engines, and `playwright install` with no argument installs all three. One mismatch trips up almost everyone. `playwright install chrome` does not install Chromium, because `chrome` is a branded channel, and Playwright doesn't install branded browsers by default. Install `chrome` and then call `pw.chromium.launch()` without a `channel` argument, and the launch fails. There's no Chromium binary on disk. A short list of the errors you're most likely to hit, and what each one means: - **`ModuleNotFoundError: No module named 'playwright'`** means the package isn't installed in the interpreter you're running. Confirm `pip` and `python` point at the same environment before reinstalling. - **`ModuleNotFoundError: No module named 'playwright.sync_api'; 'playwright' is not a package`** has a different, specific cause: a file named `playwright.py` in your working directory shadows the real package. Rename it. A folder named `playwright/` next to your script causes the same problem. - **`BrowserType.launch: Executable doesn't exist at ...`** isn't an import problem at all. The package imported fine, but the browser binary is missing. Run `playwright install chromium`. This also shows up right after upgrading Playwright, since a new release usually expects a new browser build. - Playwright reads **`PLAYWRIGHT_BROWSERS_PATH`** to find those binaries. Set it consistently for both `playwright install` and whatever process launches the browser, or the install lands somewhere the launch doesn't look. The first two are import failures. The third happens after a successful import, when Playwright tries to start a browser it can't find. Keep them separate when you're debugging, because the fixes don't overlap. ## Tip: Playwright in REPL Prototyping selectors against a live, visible browser beats guessing from saved HTML. Start the sync client manually in a Python REPL like [ipython](https://pypi.org/project/ipython/), and the browser stays open between statements so you can try selector after selector: python```python from playwright.sync_api import sync_playwright pw = sync_playwright().start() browser = pw.chromium.launch(headless=False) page = browser.new_page() page.goto("https://web-scraping.dev/products") ``` Call `pw.stop()` when you're done to close the browser and free the process. In a Jupyter notebook this pattern doesn't work at all, because the notebook already runs an event loop the sync client can't share. That has a working answer of its own: [Playwright in a Jupyter notebook](https://scrapfly.io/blog/answers/playwright-in-ipython). ## The Basics Every Playwright script launches a browser, opens a context, then opens a page inside that context. The context is where viewport size, locale, and storage state live, and it's the unit you clone when you scale to multiple parallel scrapers. The examples use purpose-built pages on `web-scraping.dev`: products for extraction, login for form interaction, testimonials for scrolling and interception, and reviews for button pagination. The selectors below match the site's current v1.3.0 markup. python```python from playwright.sync_api import sync_playwright with sync_playwright() as pw: browser = pw.chromium.launch() context = browser.new_context(viewport={"width": 1280, "height": 800}) page = context.new_page() page.goto("https://web-scraping.dev/products") print(page.title()) ``` That launches a browser, opens a context sized to a typical desktop viewport, and prints the page's title. ### Navigation and Waiting `page.goto()` resolves once the browser's `load` event fires, and on most pages that's already enough. This one included: `/products` renders its listing on the server, so the markup is complete the moment the response arrives. Plenty of real targets don't work that way. A page that fetches its data through a background call after `load` can leave `page.content()` returning a shell for a moment. Code that reads the DOM right after `goto()` hits that shell. Use the reviews page to reproduce the race. Its review cards arrive after navigation commits: python```python page.goto("https://web-scraping.dev/reviews", wait_until="commit") print(page.locator(".review").count()) # 0 page.wait_for_selector(".review") print(page.locator(".review").count()) # 20 ``` Three waiting tools cover most cases: - `page.wait_for_selector(selector)` waits for a specific element. This is the default, and it's what you should reach for first. - `page.wait_for_load_state("networkidle")` waits until the page goes 500ms without a network connection. Playwright's own docs call `networkidle` discouraged, so treat it as a fallback for when you don't know the selector yet, not a peer of `wait_for_selector`. It never settles on a page that polls in the background. - `page.wait_for_timeout(ms)` is a fixed sleep. It's for debugging, not production: too short and it's flaky, too long and it's slow. Locators also auto-wait on actions like `click()` and `fill()`, so an explicit wait before an interaction is usually redundant. The explicit wait matters when you're about to read the DOM yourself, as the next section does. For the deeper version of this question, see [how to wait for a page to load in Playwright](https://scrapfly.io/blog/answers/how-to-wait-for-page-to-load-in-playwright). ### Parsing Data Playwright locators read the DOM directly, and that's the first tool to reach for. The pattern below uses `locator.all()` rather than `element_handles()`, which Playwright's docs mark discouraged because it's racy on a changing page. One catch works the same way as navigation: `locator.all()` doesn't wait, and returns whatever matches the instant you call it. Wait for the first row with `wait_for_selector(".row.product")` first, or a slow load hands you an empty list. Verified selectors on `https://web-scraping.dev/products`: | Field | Selector | |---|---| | Product row | `.row.product` | | Title | `h3.mb-0 a` | | Product URL | `h3.mb-0 a` attribute `href` | | Description | `.short-description` | | Price | `.price` | | Thumbnail | `img.img-thumbnail` attribute `src` | python```python page.wait_for_selector(".row.product") products = [] for row in page.locator(".row.product").all(): products.append({ "title": row.locator("h3.mb-0 a").inner_text(), "url": row.locator("h3.mb-0 a").get_attribute("href"), "price": row.locator(".price").inner_text(), "description": row.locator(".short-description").inner_text(), "thumbnail": row.locator("img.img-thumbnail").get_attribute("src"), }) ``` The first record has this shape (description abridged): json```json {"title": "Box of Chocolate Candy", "url": "https://web-scraping.dev/product/1", "price": "24.99", "description": "Indulge your sweet tooth with our Box of Chocolate Candy...", "thumbnail": "https://web-scraping.dev/assets/products/orange-chocolate-box-medium-1.webp"} ``` Once the page has rendered, `page.content()` hands you the full HTML, parseable with anything. Locators are better for interacting with a page. A dedicated parser like [Parsel](https://scrapfly.io/blog/posts/guide-to-html-parsing-with-parsel-python) or [BeautifulSoup](https://scrapfly.io/blog/posts/web-scraping-with-python-beautifulsoup) is better for pulling many fields at once, especially optional ones where a missing element should return `None` instead of raising: python```python from parsel import Selector html = page.content() sel = Selector(text=html) for row in sel.css(".row.product"): print({ "title": row.css("h3.mb-0 a::text").get(), "url": row.css("h3.mb-0 a::attr(href)").get(), "price": row.css(".price::text").get(), }) # Or with BeautifulSoup, an explicit parser avoids a warning, # and reading .get("href") replaces Parsel's ::attr() syntax, # which BeautifulSoup doesn't support: from bs4 import BeautifulSoup soup = BeautifulSoup(html, "html.parser") for row in soup.select(".row.product"): link = row.select_one("h3.mb-0 a") print({"title": link.text, "url": link.get("href"), "price": row.select_one(".price").text}) ``` Both produce the same rows as the locator version. Pick a parser once you need many fields, especially optional ones. ### Clicking Buttons and Text Input The login form at `https://web-scraping.dev/login` is a cleaner form-interaction example than a search box, because you can confirm success by reading the page afterward. Verified selectors: | Element | Selector | |---|---| | Username field | `input[name="username"]` | | Password field | `input[name="password"]` | | Submit button | `form[action="/api/login"] button[type="submit"]` | | Cookie modal dismiss | `#cookie-ok` | python```python page.goto("https://web-scraping.dev/login") cookie_ok = page.locator("#cookie-ok") if cookie_ok.count() and cookie_ok.is_visible(): cookie_ok.click() page.locator('input[name="username"]').fill("user123") page.locator('input[name="password"]').fill("password") page.locator('form[action="/api/login"] button[type="submit"]').click() page.get_by_text("Logged in as User123").wait_for() print("Logged in as User123") ``` Use `fill()`, not `locator.type()`, which Playwright's docs mark as deprecated. `press_sequentially()` sends real keystrokes for the rare page with special key handling. "Type it character by character to look human" is mostly a myth worth dropping. Playwright refuses to click a locator matching more than one element, since it won't guess which one you meant, so keep selectors specific. The cookie modal above is the everyday version of a different problem, where something covers the element and you need to dismiss it first. For the general case, see [handling cookie pop-ups in Playwright](https://scrapfly.io/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-playwright). ### Scrolling and Infinite Pagination `https://web-scraping.dev/testimonials` loads more testimonials as you scroll, the same pattern real infinite-scroll pages use. Find the current last item, scroll it into view, wait, count again, and stop once the count stops growing: python```python page.goto("https://web-scraping.dev/testimonials") page.wait_for_selector(".testimonial") seen = 0 for i in range(10): # cap the loop, an unbounded scroll never stops on a genuinely endless feed items = page.locator(".testimonial") items.last.scroll_into_view_if_needed() page.wait_for_timeout(1000) count = items.count() if count == seen: break seen = count ``` Scrolling the window itself often doesn't trigger loading in a headless browser. Scrolling the last matched element into view does, because that's the element the page's own scroll listener is watching. For the scroll mechanics on their own, see [how to scroll to the bottom with Playwright](https://scrapfly.io/blog/answers/how-to-scroll-to-the-bottom-with-playwright). For pagination patterns beyond scrolling, see [Guide to List Crawling: Everything You Need to KnowComplete list crawling tutorial assess site defenses, bypass anti-bot systems, choose tools (Beautiful Soup, Playwright, Scrapfly), extract data with 6 production-ready code examples, and troubleshoot common failures.](https://scrapfly.io/blog/posts/guide-to-list-crawling) ## Advanced Functions The basics get you a working scraper. These four cover where the basics run into a wall: - JavaScript the built-in API doesn't expose - Requests you need to read or change - Bandwidth you don't need to spend - Failures you need to survive ### Evaluating Javascript `page.evaluate()` runs JavaScript inside the page and returns the result to Python. Reach for it when a built-in method doesn't do what you need. Return values must be JSON-serializable, so return plain objects and arrays, not DOM nodes. python```python page.evaluate(""" () => { const items = document.querySelectorAll('.testimonial'); items[items.length - 1].scrollIntoView({behavior: "smooth", block: "end"}); } """) ``` This is the same scroll-into-view idea from the previous section, written by hand. It's worth knowing because this is how you'd work around any built-in Playwright behavior that doesn't fit your target. ### Request and Response Intercepting Observing traffic and changing it are two different tools, and mixing them up is the most common mistake on this page. `page.on("request")` and `page.on("response")` only observe. You can't write to the objects they hand you, so setting `request.headers[...]` or `request.post_data` raises an error or silently does nothing. To change what goes over the wire, you need `page.route()` and `route.continue_()` instead. `https://web-scraping.dev/testimonials` loads more testimonials from `/api/testimonials` as you scroll. Watch for it with `page.on("request")`: python```python def log_background_call(request): if request.resource_type in ("xhr", "fetch"): print(request.method, request.url, request.resource_type) page.on("request", log_background_call) page.goto("https://web-scraping.dev/testimonials") page.wait_for_selector(".testimonial") with page.expect_request("**/api/testimonials?page=2"): page.locator(".testimonial").last.scroll_into_view_if_needed() ``` That prints `GET https://web-scraping.dev/api/testimonials?page=2 xhr`, the same call the scroll handler triggers. A plain request with no `Referer` header at all gets HTTP 422 back, `{"detail":[{"type":"missing","loc":["header","referer"],"msg":"Field required","input":null}]}`. Inside a real browser you never see that error. The page's Referrer-Policy, here the default `strict-origin-when-cross-origin`, makes the browser attach its own `Referer` to the same-page background call. That behavior is worth testing rather than assuming, and it's the actual gotcha here. `route.continue_(headers=...)` accepts a `referer` key without complaint, but tested against this Chromium build, the browser still sends its own value regardless of what you pass. Referer, like Origin and Cookie, is browser-managed, and interception doesn't hand you full control over it. Custom headers your own page sets are yours to change. The browser's own headers mostly aren't. `route.continue_(url=...)` is the change you can prove. Rewrite the page parameter the request was going to send, and the response carries different testimonials than the ones the page asked for: python```python def rewrite_page_param(route): url = route.request.url.replace("page=2", "page=3") route.continue_(url=url) page.route("**/api/testimonials**", rewrite_page_param) page.goto("https://web-scraping.dev/testimonials") page.wait_for_selector(".testimonial") with page.expect_response("**/api/testimonials?page=3") as response_info: page.locator(".testimonial").last.scroll_into_view_if_needed() print(response_info.value.url) ``` `route.continue_()` also takes `headers` and `post_data`, for adding an auth token or changing a POST body. Don't expect it to override headers the browser considers its own. For the capture-only version of this pattern, see [how to capture XHR requests in Playwright](https://scrapfly.io/blog/answers/how-to-capture-xhr-requests-playwright). ### Blocking Resources A headless browser downloads every image, font, and media file a page references, even the ones your scraper never reads. Aborting the ones you don't need cuts that waste. Playwright's complete `resource_type` list is `document`, `stylesheet`, `image`, `media`, `font`, `script`, `texttrack`, `xhr`, `fetch`, `eventsource`, `websocket`, `manifest`, and `other`. An older, wrong list floats around online with entries like `beacon` and `imageset` that never match anything, so a filter built on it silently blocks nothing. python```python BLOCK_TYPES = {"image", "media", "font"} # stylesheet and script also match, but blocking them breaks rendering # or interactivity on many pages, so they stay off by default def block_expensive_resources(route): if route.request.resource_type in BLOCK_TYPES: route.abort() else: route.continue_() page.route("**/*", block_expensive_resources) page.goto("https://web-scraping.dev/products") ``` Blocking is visible to the page. A site whose analytics or anti-bot script never loads can treat that absence as its own signal. Aborting third-party scripts by domain can get you blocked faster than downloading them would have. Blocking images and fonts is usually lower risk than blocking scripts, but it can still break pages that depend on image load events, dimensions, fonts, or visual challenges. Test the blocked configuration against the fields and interactions your scraper needs. Blocking scripts is a decision, not a free optimization. Measure the transferred bytes in your own network panel, with and without blocking, on the target you care about. For a longer walkthrough, see [how to block resources in Playwright](https://scrapfly.io/blog/answers/how-to-block-resources-in-playwright). ### Why Do Playwright Scrapers Fail and How to Retry Intelligently? The wrapper below retries `PlaywrightTimeoutError` only. HTTP status codes need a separate path because `page.goto()` returns a `Response` for 403, 429, and 5xx rather than raising a timeout. Inspect `response.status`, honor `Retry-After` on 429, retry only transient 5xx responses, and do not blanket-retry 403, 401, or 404. Exponential backoff spaces retries out instead of hammering a struggling server, `wait_time = base_delay * (2 ** attempt)`. Jitter, `wait_time * random(0.5, 1.5)`, keeps several retrying scrapers from landing on the server at once. python```python import time import random from playwright.sync_api import sync_playwright, TimeoutError as PlaywrightTimeoutError def retry_with_backoff(func, max_retries=5, base_delay=1): def wrapper(*args, **kwargs): for attempt in range(max_retries): try: return func(*args, **kwargs) except PlaywrightTimeoutError: if attempt == max_retries - 1: raise wait_time = base_delay * (2 ** attempt) time.sleep(wait_time * random.uniform(0.5, 1.5)) return None return wrapper @retry_with_backoff def scrape_page(page, url): page.goto(url, timeout=30000) page.wait_for_selector(".row.product", timeout=10000) return page.content() ``` The wrapper catches `PlaywrightTimeoutError` specifically and retries only that. A 401 or 404 means the request reached the server and got a definite answer, so retrying it wastes time instead of fixing anything. ## How to Scrape Multiple Web Pages with Playwright One page is a demo. A queue of URLs is a scraper. The same waiting and parsing patterns from above apply to each one. ### How to Loop Through Multiple URLs Reading URLs from a CSV and writing results to JSONL as you go is a solid default. A crash halfway through the run still leaves you with every result collected so far. Save this fixture as `urls.csv` beside the script: csv```csv url https://web-scraping.dev/product/1 https://web-scraping.dev/product/2 https://web-scraping.dev/product/3 https://web-scraping.dev/product/4 https://web-scraping.dev/product/5 ``` python```python import csv import json from playwright.sync_api import sync_playwright, TimeoutError as PlaywrightTimeoutError def scrape_url(page, url): try: page.goto(url, timeout=30000) page.wait_for_selector(".product-title", timeout=10000) title = page.locator(".product-title").first.inner_text() return {"url": url, "title": title, "success": True} except PlaywrightTimeoutError as e: return {"url": url, "error": f"timeout: {e}", "success": False} with open("urls.csv") as f: urls = [row["url"] for row in csv.DictReader(f)] with sync_playwright() as pw: browser = pw.chromium.launch(headless=True) page = browser.new_context().new_page() with open("results.jsonl", "w") as f: for url in urls: result = scrape_url(page, url) f.write(json.dumps(result) + "\n") browser.close() ``` Reusing one `page` object across URLs is faster than opening a fresh context per URL. Open a fresh context instead when the site tracks state you don't want carrying over, like a cart or a login session. ### How to Handle Pagination with Playwright `/reviews` paginates behind a "Load More" button. `/products?page=2` paginates through a URL parameter instead. Prefer the URL form when a site offers it, since you can parallelize and resume it from any page. python```python def scrape_all_reviews(page, start_url, max_clicks=10): page.goto(start_url) page.wait_for_selector(".review") for _ in range(max_clicks): load_more = page.locator("#page-load-more") if load_more.count() == 0 or not load_more.is_visible(): break previous_count = page.locator(".review").count() load_more.click() page.wait_for_function( "count => document.querySelectorAll('.review').length > count", arg=previous_count, ) ``` The reviews button doesn't disable itself when it runs out of pages. It disappears from the page instead, so the guard checks `is_visible()` and element count rather than `is_disabled()`. python```python from playwright.sync_api import TimeoutError as PlaywrightTimeoutError def scrape_all_pages_params(page, base_url, max_pages=50): all_titles = [] for page_num in range(1, max_pages + 1): page.goto(f"{base_url}?page={page_num}") try: page.wait_for_selector(".row.product", timeout=5000) except PlaywrightTimeoutError: break titles = page.locator("h3.mb-0 a").all_inner_texts() if not titles: break all_titles.extend(titles) return all_titles ``` Unlike the infinite scroll covered earlier, both of these stop on an explicit signal, not a scroll position. ### How to Scrape Pages Concurrently with Asyncio Concurrency helps because a page spends most of its time waiting on the network, not computing. It stops helping once you saturate memory, CPU, or the target's tolerance for simultaneous connections. A semaphore caps how many browser contexts run at once, so you don't open more than your machine or the target can handle: python```python import asyncio from playwright.async_api import async_playwright, TimeoutError as PlaywrightTimeoutError async def scrape_url_async(browser, url, semaphore): async with semaphore: context = await browser.new_context(viewport={"width": 1280, "height": 800}) page = await context.new_page() try: await page.goto(url, timeout=30000) await page.wait_for_selector(".product-title", timeout=10000) title = await page.locator(".product-title").first.inner_text() return {"url": url, "title": title, "success": True} except PlaywrightTimeoutError as e: return {"url": url, "error": f"timeout: {e}", "success": False} finally: await context.close() async def scrape_multiple_concurrent(urls, max_concurrent=5): async with async_playwright() as pw: browser = await pw.chromium.launch(headless=True) semaphore = asyncio.Semaphore(max_concurrent) results = await asyncio.gather(*[ scrape_url_async(browser, url, semaphore) for url in urls ]) await browser.close() return results urls = [f"https://web-scraping.dev/product/{i}" for i in range(1, 6)] results = asyncio.run(scrape_multiple_concurrent(urls, max_concurrent=3)) ``` If you're crawling several domains at once, wrap each domain's own `asyncio.Semaphore` around its URLs. That way, one slow or strict domain doesn't steal every open slot from the others. Start with one concurrent context, record throughput, memory, CPU, and error rate, then raise the limit one step at a time until throughput flattens or failures increase. A browser context holds real memory for as long as it's open. The ceiling is something you measure on your own target, not a number you copy from elsewhere. `ThreadPoolExecutor` can overlap independent synchronous I/O, but Playwright requires a separate Playwright instance per thread because its API is not thread-safe. Threads do not speed up CPU-heavy Python parsing under the GIL; use a process pool for that work. For concurrent Playwright pages, the async client is the simpler default. Scrapfly #### Need a cloud browser for scraping? Run headless browsers at scale with Scrapfly Cloud Browser — no infrastructure to manage. [Try Free →](https://scrapfly.io/register)## How Fast and How Expensive Is a Playwright Scraper? The biggest lever isn't a Playwright setting at all. Check whether you need a browser. The interception section above showed you how to find a page's background API. Calling that endpoint directly is faster than rendering the whole page, by a wide margin. Check the background API before optimizing browser settings. Once you do need the browser, the levers that move the needle, in order: - Block images, fonts, and media. The largest bandwidth win, and the lowest risk. - Reuse the browser process across URLs instead of launching a fresh one per URL. Launching is the expensive part. - Wait for a selector rather than `networkidle`, which makes you wait for trackers you don't care about. - Run headless. Headless and headful are different code paths, and a scraper can behave differently between them, so debug headful and run headless. - Raise concurrency until throughput stops improving. | Lever | Bandwidth effect | Speed effect | Risk of breaking the page | |---|---|---|---| | Skip the browser | n/a | Largest | Low, if the API is stable | | Block resources | Large | Medium | Low for images/fonts, higher for scripts | | Reuse process | None | Medium | Low | | Selector waits | None | Small to medium | Low | | Headless | Small | Small | Low, but verify headful first | | Raise concurrency | None | Large, until it isn't | Medium, watch memory and rate limits | A browser holds memory for its entire lifetime, and that's the real constraint on how many you can run at once. That cost, and the blocking-visibility tradeoff from the resource section, are what decide whether to keep running your own browsers. The next section picks up where that decision leads. ## Avoiding Blocking A real browser clears the low bar. That's why Playwright works on plenty of sites a plain HTTP client can't touch. It doesn't clear the high bar, because a site can observe the automation itself, on top of the requests it sends. ### What are Honeypot Traps and How to Avoid Them? Sites hide [honeypot elements](https://scrapfly.io/blog/posts/what-are-honeypots-and-how-to-avoid-them) that no visitor would ever see, so clicking one flags whatever clicked it as code. Check visibility before you interact with anything you didn't target deliberately: python```python load_more = page.locator("#page-load-more") if load_more.is_visible(): load_more.click() ``` This skips anything hidden with CSS, the exact elements a honeypot depends on you clicking blindly. ### How to Configure a Proxy Pass one `proxy` dict to `launch()`, with `server` and, if the proxy needs it, `username` and `password` in the same dict: python```python browser = pw.chromium.launch( headless=True, proxy={"server": "http://12.34.56.78:8080", "username": "user", "password": "pass"}, ) ``` Datacenter proxies are cheaper and easier to detect. Residential proxies are the opposite. Rotation, cycling through a pool on each request, is a behavior you add on top of either kind, not a third category of proxy. For rotation strategy beyond the syntax above, see [How to Rotate Proxies in Web ScrapingIn this article we explore proxy rotation. How does it affect web scraping success and blocking rates and how can we smartly distribute our traffic through a pool of proxies for the best results.](https://scrapfly.io/blog/posts/how-to-rotate-proxies-in-web-scraping) ### Advanced Detection Vectors A handful of signals combine to flag automation: - `navigator.webdriver` reports `true` in a default Playwright browser, which makes it the first thing most detection scripts check. - Canvas and WebGL rendering differences produce a signature that's stable per machine. - A headless build behaves differently from a headful one in ways a page can read. - Identical timing between actions across a session reads as a machine, not a person. Cloudflare, DataDome, Akamai, PerimeterX, and Kasada combine signals like these into a single score. TLS fingerprinting isn't one of them, in this specific comparison. A Playwright-driven Chromium presents Chromium's own TLS stack. TLS separates a raw HTTP client from any browser, not an automated browser from a manual one. [Our stealth guide](https://scrapfly.io/blog/posts/playwright-stealth-bypass-bot-detection) covers these signals in depth, along with the Python and Node stealth libraries and where each stops working. For the fingerprinting mechanics themselves, see [How Javascript is Used to Block Web Scrapers? In-Depth GuideIntroduction to how javascript is used to detect web scrapers. What's in javascript fingerprint and how to correctly spoof it for web scraping.](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-javascript) ## How to Connect Playwright to a Remote Browser With connect\_over\_cdp `chromium.connect_over_cdp(ws_url)` attaches Playwright to a browser running somewhere else, instead of one on your own machine. Everything after that call is ordinary Playwright, the same locators, waits, and interception from earlier. That's the whole point, and it's why this path is cheap to try. It matters when your own browser setup costs more than the data you're getting from it. It also matters when the target blocks you regardless of what you patch locally. If your scraper already runs fine on your own machines, none of this changes anything for you. Scrapfly's Cloud Browser is a concrete instance you can run yourself, against the same product page used throughout this article. It swaps `chromium.launch()` for `chromium.connect_over_cdp()`, reuses the browser's existing context and page instead of creating new ones, and closes the browser in a `finally` block: python```python from playwright.sync_api import sync_playwright API_KEY = "" BROWSER_WS = f"wss://browser.scrapfly.io?api_key={API_KEY}&proxy_pool=public_datacenter_pool&os=linux" with sync_playwright() as pw: browser = pw.chromium.connect_over_cdp(BROWSER_WS) try: context = browser.contexts[0] page = context.pages[0] if context.pages else context.new_page() page.goto("https://web-scraping.dev/products") print(page.title()) finally: browser.close() ``` The connection string carries the parameters that matter for scraping. `proxy_pool` picks datacenter or residential, `country` sets geo-targeting, and `os`/`browser_brand` set the fingerprint profile. Add `session` with `auto_close=false` when you need to reconnect to the same browser later, and `timeout` for the session ceiling. One parameter ties straight back to the Blocking Resources section. The `block_images`, `block_styles`, `block_fonts`, `block_media`, and `blacklist` parameters **stub** requests instead of aborting them. The browser gets a valid but empty response, like a blank single-pixel image, instead of a failed request. That saves the bandwidth without the aborted-request pattern some anti-bot systems watch for, which `route.abort()` can't avoid. Two limitations are worth knowing before you hit them. The Cloud Browser disables page console instrumentation as part of its stealth hardening, so `page.set_content()` times out even though the HTML gets applied. `page.on("console")` never receives page output either. Build documents with `page.evaluate()` instead. With the default `auto_close=true`, `browser.close()` ends the one-shot session. With `session=...&auto_close=false`, Playwright's `browser.close()` only disconnects; the remote browser keeps running and billing until its timeout or an explicit `POST /session/{session_id}/stop`. Use `try`/`finally` for the disconnect, then stop a named session when you are finished with it. None of this is Python-specific. The endpoint is a CDP WebSocket, so Playwright's Node client and Puppeteer connect to it the same way, with the same connection string. ## Where to Go Next These guides continue from the specific problem you have: - **Stop getting blocked:** [Playwright stealth and bot detection](https://scrapfly.io/blog/posts/playwright-stealth-bypass-bot-detection) - **Copy a working snippet:** [Playwright examples in JavaScript and Python](https://scrapfly.io/blog/posts/playwright-examples-javascript-and-python) - **Fix a waiting bug:** [wait for a page to load](https://scrapfly.io/blog/answers/how-to-wait-for-page-to-load-in-playwright) - **Run Playwright in a notebook:** [Playwright in IPython](https://scrapfly.io/blog/answers/playwright-in-ipython) - **Use Playwright inside Scrapy:** [Scrapy with Playwright](https://scrapfly.io/blog/posts/how-to-use-scrapy-with-playwright) - **Write it in JavaScript instead:** [Playwright with JavaScript](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-javascript) - **Compare Playwright with an AI browser agent:** [Browser Use vs Playwright](https://scrapfly.io/blog/posts/browser-use-vs-playwright) - **Load a browser extension:** [browser extensions with Playwright](https://scrapfly.io/blog/posts/how-to-use-browser-extensions-with-playwright-puppeteer-and-selenium) - **Single-task recipes:** [screenshots](https://scrapfly.io/blog/answers/how-to-take-screenshot-with-playwright), [cookies](https://scrapfly.io/blog/answers/how-to-save-and-load-cookies-in-playwright), [file downloads](https://scrapfly.io/blog/answers/how-to-download-file-with-playwright), [element checks](https://scrapfly.io/blog/answers/how-to-check-for-element-in-playwright), [XPath](https://scrapfly.io/blog/answers/how-to-find-elements-by-xpath-in-playwright), and [CSS selectors](https://scrapfly.io/blog/answers/how-to-find-elements-by-css-selectors-in-playwright). ## FAQ Is Playwright good for web scraping?Yes, for pages that build their content with JavaScript, where an HTTP client returns an empty shell. For server-rendered HTML, an HTTP client plus a parser is faster and cheaper, so check what the page returns before reaching for a browser. Do Anti-Bot Systems Detect Playwright?Yes. A default Playwright browser reports `navigator.webdriver` as `true` and carries other automation signals that anti-bot systems read. See the stealth guide linked above for what that takes to change. Should I use Chromium, Firefox, or WebKit for scraping?Chromium is the default, with the widest tooling support, including CDP-based remote connection. Firefox is worth trying when a target treats Chromium traffic more harshly. WebKit mainly matters for checking Safari-specific rendering. How do I use a proxy with Playwright?Pass one `proxy` dict to `launch()`, with `server` and, if needed, `username` and `password` in that same dict. See the proxy configuration section above for the exact syntax. Playwright or BeautifulSoup: which do you need?They do different jobs. Playwright fetches and renders a page, and BeautifulSoup parses HTML you already have. Render with Playwright, pass `page.content()` to BeautifulSoup, or skip Playwright entirely when the raw HTML already has your data. Can I use Playwright with JavaScript instead of Python?Yes, the API is close to identical between the two. See our Playwright JavaScript guide, linked earlier, for the full version. How many pages can I scrape in parallel with Playwright?Start with one concurrent context and raise the limit while measuring throughput, memory, CPU, and error rate. Stop when throughput flattens or failures increase. The ceiling depends on the page, so measure it on your own target instead of copying a figure. What retry strategy works for 403 and 429 errors?The wrapper above retries Playwright timeouts only. To retry HTTP responses, inspect the `Response` returned by `page.goto()`: honor `Retry-After` on 429, retry only transient 5xx statuses, and do not blanket-retry 403, 401, or 404. A repeated 403 needs a blocking fix, not a longer delay. ## Summary You now have a Playwright scraper that waits for content correctly and extracts it with locators or a parser handoff. It fills out forms, scrolls, paginates, and reads and rewrites its own requests. Blocking unneeded resources reduces transferred bytes. Concurrency raises total throughput until CPU, memory, or the target becomes the bottleneck. The timeout wrapper retries slow navigations with backoff. HTTP-status retries need separate handling based on the `Response` returned by `page.goto()`. Two limits are worth carrying forward. A browser is expensive per page, so confirm you need one before reaching for Playwright by default. A real browser isn't invisible, either. It clears bots a plain HTTP client can't get past. But the automation itself stays observable to a site looking for it. From here, the Where to Go Next section above routes to your specific problem, whether that's stealth or a single-task recipe. For moving off your own setup entirely, there's Scrapfly's Cloud Browser, covered above. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [What is Playwright?](#what-is-playwright) - [Setup](#setup) - [Tip: Playwright in REPL](#tip-playwright-in-repl) - [The Basics](#the-basics) - [Navigation and Waiting](#navigation-and-waiting) - [Parsing Data](#parsing-data) - [Clicking Buttons and Text Input](#clicking-buttons-and-text-input) - [Scrolling and Infinite Pagination](#scrolling-and-infinite-pagination) - [Advanced Functions](#advanced-functions) - [Evaluating Javascript](#evaluating-javascript) - [Request and Response Intercepting](#request-and-response-intercepting) - [Blocking Resources](#blocking-resources) - [Why Do Playwright Scrapers Fail and How to Retry Intelligently?](#why-do-playwright-scrapers-fail-and-how-to-retry-intelligently) - [How to Scrape Multiple Web Pages with Playwright](#how-to-scrape-multiple-web-pages-with-playwright) - [How to Loop Through Multiple URLs](#how-to-loop-through-multiple-urls) - [How to Handle Pagination with Playwright](#how-to-handle-pagination-with-playwright) - [How to Scrape Pages Concurrently with Asyncio](#how-to-scrape-pages-concurrently-with-asyncio) - [How Fast and How Expensive Is a Playwright Scraper?](#how-fast-and-how-expensive-is-a-playwright-scraper) - [Avoiding Blocking](#avoiding-blocking) - [How to Connect Playwright to a Remote Browser With connect\_over\_cdp](#how-to-connect-playwright-to-a-remote-browser-with-connect-over-cdp) - [Where to Go Next](#where-to-go-next) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ python nodejs ### Concurrency vs Parallelism Learn the key differences between Concurrency and Parallelism and how to leverage them in Python and JavaScript to optim... ](https://scrapfly.io/blog/posts/concurrency-vs-parallelism) [ http python ### Web Scraping with Python Introduction tutorial to web scraping with Python. How to collect and parse public data. Challenges, best practices and ... ](https://scrapfly.io/blog/posts/web-scraping-with-python) [ python headless-browser ### Web Scraping with Selenium and Python Introduction to web scraping dynamic javascript powered websites and web apps using Selenium browser automation library ... ](https://scrapfly.io/blog/posts/web-scraping-with-selenium-and-python) ## Related Questions - [ Q How to block resources in Playwright and Python? ](https://scrapfly.io/blog/answers/how-to-block-resources-in-playwright) - [ Q How to capture background requests and responses in Playwright? ](https://scrapfly.io/blog/answers/how-to-capture-xhr-requests-playwright) Run headless browsers at scale, **1,000 free credits** [Start Free](https://scrapfly.io/register)