[Blog](https://scrapfly.io/blog) / [headless-browser](https://scrapfly.io/blog/tag/headless-browser) / [Scrapy Selenium Guide: JavaScript Pages With Selenium 4](https://scrapfly.io/blog/posts/web-scraping-dynamic-web-pages-with-scrapy-selenium) # Scrapy Selenium Guide: JavaScript Pages With Selenium 4 by [Mazen Ramadan](https://scrapfly.io/blog/author/mazen) Aug 27, 2026 22 min read [\#headless-browser](https://scrapfly.io/blog/tag/headless-browser) [\#python](https://scrapfly.io/blog/tag/python) [\#scrapy](https://scrapfly.io/blog/tag/scrapy) [\#selenium](https://scrapfly.io/blog/tag/selenium) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-dynamic-web-pages-with-scrapy-selenium "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-dynamic-web-pages-with-scrapy-selenium&text=Scrapy%20Selenium%20Guide%3A%20JavaScript%20Pages%20With%20Selenium%204 "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-dynamic-web-pages-with-scrapy-selenium "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-dynamic-web-pages-with-scrapy-selenium) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-dynamic-web-pages-with-scrapy-selenium) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-dynamic-web-pages-with-scrapy-selenium) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-dynamic-web-pages-with-scrapy-selenium) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-dynamic-web-pages-with-scrapy-selenium) `pip install scrapy-selenium` still succeeds. The first crawl then stops with `TypeError: WebDriver.__init__() got an unexpected keyword argument 'executable_path'`. That package was published in January 2019, and Selenium 4 later dropped the argument it passes. A pinned stack does still run. This guide sets one up and puts it through rendering, scrolling, a login form, and screenshots. It also marks where that route stops being the right pick. [Web Scraping With Scrapy: The Complete Guide in 2026Build and run a Scrapy 2.18 project with async start(), pagination, selector tests, item validation, pipelines, and JSON Lines export.](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) ## Key Takeaways - **`scrapy-selenium` 0.0.7 is from 2019**, so use the `scrapy-selenium4` 1.0.0 fork instead. - **Wait on observable conditions**, not fixed sleeps, whenever the page state is visible. - **Bound every scroll loop** with a round cap and a check that the item count grew. - **One driver serves the whole crawl**, so keep Selenium concurrency at one request. - **Rendering is not a bypass**, and the target can still challenge or rate-limit you. - **Pick Playwright for new Scrapy browser work**, and keep Selenium for existing crawlers. - **For rendered HTML at scale**, a managed endpoint runs the browsers so you only parse. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## Does Scrapy Selenium still work with Selenium 4? The original [scrapy-selenium](https://pypi.org/project/scrapy-selenium/) path does not. Its middleware builds the driver with a Selenium 3 constructor, which Selenium 4 rejects. The fork published as [scrapy-selenium4](https://pypi.org/project/scrapy-selenium4/) does run on Scrapy 2.18 and Selenium 4, against a local browser and driver pair. This is the exact stack the guide was tested on, with the limit attached to each piece: | Component | Article version | Status in this guide | Important limit | |---|---|---|---| | Python | 3.14 | Current example runtime | Scrapy 2.18 needs Python 3.10 or newer. | | Scrapy | 2.18.0 | Current | Spiders use `async def start()`. | | Selenium | 4.47.0 | Current example runtime | Browser and driver majors must match. | | `scrapy-selenium` | 0.0.7 | Do not use in the example | Published 2019-01-24 on a Selenium 3 constructor path. | | `scrapy-selenium4` | 1.0.0 | Compatibility path | Small project, and its middleware already logs a Scrapy deprecation warning. | | Chrome and ChromeDriver | Matching majors | Local browser path | Do not hardcode a reader-specific path. | The 0.0.7 release is a fact about the package index, not a judgement about the maintainers. It predates the Selenium release this article targets, so its code path cannot work here. That settles what runs. The next question is whether it's the right stack to build on. ## When should you keep Scrapy Selenium instead of moving to Playwright? Keep it when the cost of leaving is real. Move when you're starting fresh. Three cases cover almost every Scrapy project that needs a browser: - **Keep Selenium** when a crawler already leans on WebDriver actions, Selenium locators, or a [Selenium Grid](https://scrapfly.io/blog/posts/intro-to-web-scraping-using-selenium-grid) workflow you don't want to rebuild. - **Choose [Scrapy Playwright](https://scrapfly.io/blog/posts/how-to-use-scrapy-with-playwright)** for a new browser integration, current browser-context features, and a more active Scrapy-side project. - **Use plain Scrapy** when the data already sits in the HTTP response or in a background API the page calls. That third case is worth checking first, every time. A browser costs seconds per page and a lot of memory, and plenty of pages hand over their data without one. If a browser is required, the setup below is the shortest path that works. ## How do you install Scrapy 2.18 with Selenium 4? You need three pinned packages and a fresh environment. Nothing here edits an installed package, and nothing downloads a driver behind your back. ### How do you create the pinned Python environment? Create the environment, install the three pins, then print what landed: bash```bash python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install "Scrapy==2.18.0" "selenium==4.47.0" "scrapy-selenium4==1.0.0" python -c "import scrapy, selenium; print(scrapy.__version__, selenium.__version__)" scrapy startproject reviewscraper . ``` The version line prints `2.18.0 4.47.0` on a clean install. The last command scaffolds a standard Scrapy project in the current directory, which is where the settings and spiders below go. ### How do Chromium and ChromeDriver versions stay compatible? For a manually configured local driver, the browser and driver major versions have to match. A Chrome 152 install needs a ChromeDriver 152 build. A mismatch shows up as `session not created` the moment the middleware starts. You don't have to resolve those paths by hand. [Selenium Manager](https://www.selenium.dev/documentation/selenium_manager/) ships inside Selenium 4 and reports the driver and browser it would use here. The catch is that `scrapy-selenium4` never calls Selenium Manager itself. Its middleware needs an explicit executable path in the settings, so you resolve the path first and hand it over. The next section does exactly that. ## How do you configure scrapy-selenium4 in a Scrapy project? One middleware entry and a handful of `SELENIUM_*` settings. The middleware reads them in `from_crawler` and builds a single driver. It raises `NotConfigured` unless `SELENIUM_DRIVER_NAME` is set alongside either `SELENIUM_DRIVER_EXECUTABLE_PATH` or a remote `SELENIUM_COMMAND_EXECUTOR`. ### Which scrapy-selenium4 settings belong in settings.py? Append this to the generated `reviewscraper/settings.py`: python```python import os from selenium.webdriver.common.selenium_manager import SeleniumManager # Resolve local Chrome and ChromeDriver once, at startup. _paths = SeleniumManager().binary_paths(["--browser", "chrome"]) SELENIUM_DRIVER_NAME = "chrome" SELENIUM_DRIVER_EXECUTABLE_PATH = os.environ.get("CHROMEDRIVER_PATH", _paths["driver_path"]) SELENIUM_BROWSER_EXECUTABLE_PATH = os.environ.get("CHROME_PATH", _paths["browser_path"]) SELENIUM_DRIVER_ARGUMENTS = ["--headless=new", "--window-size=1920,1080"] DOWNLOADER_MIDDLEWARES = { "scrapy_selenium4.SeleniumMiddleware": 800, } CONCURRENT_REQUESTS = 1 ``` Both executable paths fall back to an environment variable, so nothing reader-specific is hardcoded. `CONCURRENT_REQUESTS = 1` matters more than it looks, because the middleware drives one browser synchronously for every request it handles. Container images usually need two more flags, `--no-sandbox` and `--disable-dev-shm-usage`. The first turns off the Chrome sandbox, which removes a real isolation boundary, so add it only where the container is that boundary. The second moves shared memory to disk and works around the small `/dev/shm` many images ship. Neither flag belongs in a workstation configuration. Expect one warning on Scrapy 2.18. The middleware still declares `process_request(request, spider)`, and Scrapy logs a `ScrapyDeprecationWarning` for it on every run. The crawl finishes anyway. A later Scrapy release will need this integration updated. With the middleware wired in, the first spider can ask for a rendered page. ## How do you wait for JavaScript content in Scrapy Selenium? By handing the wait to Selenium before Scrapy ever sees a response. A `SeleniumRequest` takes a condition, the middleware blocks on it, and your callback runs against HTML that already contains the data. ### How do you request a rendered page with async start()? The target is `https://web-scraping.dev/reviews`, a page on [web-scraping.dev](https://web-scraping.dev/) that fetches its reviews from a GraphQL endpoint after load. A plain HTTP request returns the template with zero reviews in it. Save this as `reviewscraper/spiders/reviews.py`: python```python import scrapy from scrapy_selenium4 import SeleniumRequest from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC class ReviewsSpider(scrapy.Spider): name = "reviews" allowed_domains = ["web-scraping.dev"] async def start(self): yield SeleniumRequest( url="https://web-scraping.dev/reviews", callback=self.parse, wait_time=20, wait_until=EC.presence_of_element_located( (By.CSS_SELECTOR, "[data-testid='review']") ), ) def parse(self, response): reviews = response.css("[data-testid='review']") yield { "title": response.css("title::text").get(), "reviews_rendered": len(reviews), "first_review": reviews.css("[data-testid='review-text']::text").get(), } ``` `async def start()` is the entry point Scrapy 2.13 introduced in place of `start_requests()`. The `wait_until` condition is a standard Selenium [expected condition](https://www.selenium.dev/documentation/webdriver/waits/), and `wait_time` caps it at 20 seconds. ### How do you parse the Selenium HTMLResponse with Scrapy selectors? Nothing special happens in the callback. The middleware takes the final `driver.page_source`, wraps it in a Scrapy `HtmlResponse`, and passes it through the normal pipeline. Your usual [CSS selectors](https://scrapfly.io/blog/posts/parsing-html-with-css) work unchanged. Run it and write the item to a file: bash```bash scrapy crawl reviews -O reviews.json ``` json```json [ {"title": "web-scraping.dev latest product reviews | graphql mock website", "reviews_rendered": 20, "first_review": "Unique flavor and great energy boost. It's the perfect gamer's drink!"} ] ``` Twenty reviews came back because that's one GraphQL page. The exact count moves with the target, so read it as evidence that rendering worked, not as a number to assert about the site. That item came from a single rendered view. Pages that append content as you scroll need the live driver, not the response Scrapy handed you. ## How do you scroll with Selenium inside a Scrapy callback? Through the driver that the middleware puts in the request meta. The response is a snapshot from before your callback ran, so anything you trigger by scrolling has to be read back off the live driver. The reviews page pages behind a button, so switch to `https://web-scraping.dev/testimonials` for this one. It appends the next batch whenever the last card is revealed, which is a real scroll trigger. Request it the same way, waiting on the first card: python```python class TestimonialsSpider(scrapy.Spider): name = "testimonials" allowed_domains = ["web-scraping.dev"] SCROLL_ROUNDS = 8 async def start(self): yield SeleniumRequest( url="https://web-scraping.dev/testimonials", callback=self.parse, wait_time=20, wait_until=EC.presence_of_element_located((By.CSS_SELECTOR, ".testimonial")), ) ``` The callback below adds three imports to the spider file: `Selector` from `scrapy.selector`, `WebDriverWait` from `selenium.webdriver.support.wait`, and `TimeoutException` from `selenium.common.exceptions`. `By` is already imported above. python```python def parse(self, response): driver = response.request.meta["driver"] wait = WebDriverWait(driver, timeout=10) seen = len(driver.find_elements(By.CSS_SELECTOR, ".testimonial")) for _ in range(self.SCROLL_ROUNDS): driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") try: wait.until(lambda d: len(d.find_elements(By.CSS_SELECTOR, ".testimonial")) > seen) except TimeoutException: break seen = len(driver.find_elements(By.CSS_SELECTOR, ".testimonial")) for card in Selector(text=driver.page_source).css(".testimonial"): yield { "rating": len(card.css("span.rating > svg")), "text": card.css("p.text::text").get(), } ``` Two guards do the work here. `SCROLL_ROUNDS` caps the loop so a stuck page can't spin forever, and the wait only accepts a card count that grew. A run against the live page yields 60 items, then stops on the timeout because there was nothing left to append. Scrolling reads a page. Logging in changes it, and that needs a few more conditions. Scrapfly #### Need a cloud browser for scraping? Run headless browsers at scale with Scrapfly Cloud Browser — no infrastructure to manage. [Try Free →](https://scrapfly.io/register)## How do you click buttons and submit forms with Scrapy Selenium? It works through the same driver handle, with a wait after every state change. The public login page at `https://web-scraping.dev/login?cookies=` opens with a cookie modal, which makes it a decent shape to copy. Keep the credentials out of the file. The sample account on that page is `user123` / `password`, and the spider reads both from the environment. This callback needs `os`, `By`, `EC`, and `WebDriverWait` imported at the top of the file: python```python async def start(self): yield SeleniumRequest( url="https://web-scraping.dev/login?cookies=", callback=self.parse, wait_time=20, wait_until=EC.element_to_be_clickable((By.CSS_SELECTOR, "button#cookie-ok")), ) ``` python```python def parse(self, response): driver = response.request.meta["driver"] wait = WebDriverWait(driver, timeout=10) driver.find_element(By.CSS_SELECTOR, "button#cookie-ok").click() wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR, "button#cookie-ok"))) driver.find_element(By.CSS_SELECTOR, "input[name='username']").send_keys(os.environ["WSD_USERNAME"]) driver.find_element(By.CSS_SELECTOR, "input[name='password']").send_keys(os.environ["WSD_PASSWORD"]) driver.find_element(By.CSS_SELECTOR, "button[type='submit']").click() wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "div#secret-message"))) yield { "logged_in": True, "secret": driver.find_element(By.CSS_SELECTOR, "div#secret-message").text, } ``` The request that reaches this callback waits on `element_to_be_clickable` for the cookie button, so the modal is guaranteed to be there. After the click, the spider waits for it to disappear before touching the form underneath. Nothing gets yielded on faith. The final wait on `div#secret-message` is what proves the session is authenticated, and the item carries that text as evidence: json```json [ {"logged_in": true, "secret": "🤫"} ] ``` Both flows above leave a browser running. Screenshots and shutdown close that loop. ## How do you take screenshots and always close Selenium resources? The request takes a `screenshot=True` flag, and the package owns the driver lifecycle. Both are short, and both have an edge worth knowing. ### How does a SeleniumRequest return screenshot bytes? The middleware captures the PNG after the wait resolves and drops the raw bytes into `response.meta`: python```python async def start(self): yield SeleniumRequest( url="https://web-scraping.dev/reviews", callback=self.parse, screenshot=True, wait_time=20, wait_until=EC.presence_of_element_located((By.CSS_SELECTOR, "[data-testid='review']")), ) def parse(self, response): with open("reviews.png", "wb") as image: image.write(response.meta["screenshot"]) ``` That run wrote an 81 KB PNG, 1902 by 928 pixels, because the viewport comes out smaller than the `--window-size=1920,1080` in the settings. A screenshot tells you what the browser painted, and it says nothing about whether your selectors matched, so keep asserting on parsed values too. ### Who closes the Selenium driver in scrapy-selenium4? The middleware connects its own `spider_closed` handler in `from_crawler` and calls `driver.quit()` there. A normal crawl, a `CloseSpider`, or a first keyboard interrupt all reach that handler. A second Ctrl-C forces the shutdown and skips it. A hard process kill does not. `SIGKILL` skips the signal entirely and leaves Chrome and ChromeDriver running, so check for orphan processes after a crash. Creating a second driver inside a callback leaks the same way, since only the middleware's own instance is ever closed. That covers the runs that finish cleanly. Failed runs announce themselves differently. [Web Scraping with Selenium and PythonIntroduction to web scraping dynamic javascript powered websites and web apps using Selenium browser automation library and Python.](https://scrapfly.io/blog/posts/web-scraping-with-selenium-and-python) ## What breaks in Scrapy Selenium and how do you debug it? Almost every failed run shows up as one of six symptoms, each with a check that narrows it quickly: | Symptom | Cause to check | Bounded fix | |---|---|---| | `unexpected keyword argument 'executable_path'` | The old `scrapy-selenium` package, or Selenium 3 constructor code | Install `scrapy-selenium4` and pass `Service(executable_path=...)` | | `session not created` | Browser and driver majors differ, bad binary path, or a rejected launch flag | Print both versions, then drop custom flags one at a time | | Wait timeout | Wrong selector, or the state never appeared | Re-run headed and confirm the selector in DevTools | | Empty `HtmlResponse` | The callback ran before the state existed, or the target served a challenge | Add a `wait_until` and log `driver.page_source` length | | Crawl stalls | Concurrency above one against a synchronous browser | Set `CONCURRENT_REQUESTS = 1` | | Orphan browser processes | The process died before `spider_closed` | Kill leftovers, and never open a driver inside a callback | None of these are fixed by editing files in `site-packages`. That change is invisible to your repository and disappears on the next install, so treat any advice to patch installed source as a warning sign. One symptom deserves its own section, because the fix people reach for is usually the wrong one. ## Can Scrapy Selenium bypass bot protection? No. Selenium gives you a browser and an interaction API. That is all it gives you. The target can still fingerprint the automation, throttle your IP, or answer with a challenge page. None of that changes because the JavaScript executed. When a rendered page comes back empty, the problem is usually detection rather than timing. Start by reading what the response contains. [Anti-blocking mechanisms](https://scrapfly.io/blog/posts/how-to-bypass-anti-bot-protection-when-web-scraping) covers the signals worth reading before you touch the spider again. Rendering and detection are separate problems, and you may not want to own either one. ## When should Selenium work use a standalone Scrapfly product? When the browser is a means to an end and not the thing you're building. If the job is fetching a rendered page through clean proxies, a managed endpoint removes the whole layer this guide configured. ScrapFly's [Web Scraping API](https://scrapfly.io/products/web-scraping-api) is a single HTTP endpoint for collecting web data at scale, with a **99.99% success rate** across **130M+ proxies in 190+ countries**. - [Anti-Scraping Protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - automatically defeats Cloudflare, DataDome, PerimeterX, Akamai, and 90+ other bot systems. - [Smart proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) - residential and datacenter pools with country and ASN level geo-targeting. - [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - render SPAs and dynamic pages through real cloud browsers. - [Browser automation scenarios](https://scrapfly.io/docs/scrape-api/javascript-scenario) - scroll, click, fill forms, and wait for elements without managing a browser fleet. - [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - return pages as HTML, JSON, clean text, or LLM ready Markdown. - [Session management](https://scrapfly.io/docs/scrape-api/session) - keep cookies, headers, and IPs consistent across multi step flows. - [Smart caching](https://scrapfly.io/docs/scrape-api/getting-started#api_param_cache) - cache successful responses to cut cost on repeat scraping jobs. - [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Rust](https://scrapfly.io/docs/sdk/rust), [Scrapy](https://scrapfly.io/docs/sdk/scrapy), and [no-code integrations](https://scrapfly.io/docs/integration/getting-started) including [Make](https://scrapfly.io/integration/make), [n8n](https://scrapfly.io/integration/n8n), [Zapier](https://scrapfly.io/integration/zapier), [LangChain](https://scrapfly.io/integration/langchain), and [LlamaIndex](https://scrapfly.io/integration/llamaindex). For long interactive sessions that need a driver you keep, [Cloud Browser](https://scrapfly.io/products/cloud-browser-api) runs managed browsers you connect to over a WebSocket. Note the boundary before planning around it. That surface speaks the Chrome DevTools Protocol, and the [Selenium guidance](https://scrapfly.io/docs/cloud-browser-api/selenium) is explicit that Selenium has no native transport for a remote CDP WebSocket URL. ### Web Scraping API Scrape any website with our powerful API. Anti-bot bypass, JavaScript rendering, and rotating proxies built-in. [Try Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started) ## FAQ Can one Selenium browser be shared across Scrapy requests?`scrapy-selenium4` already shares one driver across the whole crawl, which is why `CONCURRENT_REQUESTS = 1` belongs in the settings. WebDriver carries mutable cookies and navigation state, so pooling drivers is out of scope here. What replaced start\_requests() in Scrapy spiders?`async def start()` is the current entry point for the first requests a spider sends. It arrived in Scrapy 2.13, and older `start_requests()` examples need porting before they run on 2.18. Is scraping with Scrapy Selenium legal?Collecting publicly available data is generally legal, and the rules tighten around personal data, copyrighted material, and content behind a login. Check the target's terms and your local law before you crawl. Are there other ways to render JavaScript in Scrapy?Yes. [Scrapy Splash](https://scrapfly.io/blog/posts/web-scraping-with-scrapy-splash) runs a rendering service alongside your spiders, and the Playwright integration linked earlier is the more current browser route. Can Selenium Manager install Chrome as well as ChromeDriver?Yes, Chrome arrived in Selenium 4.11, Firefox in 4.12, and Edge in 4.14, each cached alongside its driver. Selenium Manager fetches them from vendor endpoints, so an offline image still needs the binaries supplied. ## Summary The Selenium 4 path through Scrapy works, and it is narrow. Three pins, one middleware entry, an explicit driver path, and a single browser serving the crawl. Everything else here is the discipline that keeps it from hanging. Wait on conditions you can observe rather than sleeping. Cap the loops, and never yield a state you haven't checked. Treat that as a maintenance route. It's the right call when a crawler already speaks WebDriver, and the wrong one when you're picking a browser layer today. For a new Scrapy project, start with Playwright instead. When the browser itself is the overhead rather than the point, hand rendering to a managed endpoint. Keep your spider focused on the data. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [Does Scrapy Selenium still work with Selenium 4?](#does-scrapy-selenium-still-work-with-selenium-4) - [When should you keep Scrapy Selenium instead of moving to Playwright?](#when-should-you-keep-scrapy-selenium-instead-of-moving-to-playwright) - [How do you install Scrapy 2.18 with Selenium 4?](#how-do-you-install-scrapy-2-18-with-selenium-4) - [How do you create the pinned Python environment?](#how-do-you-create-the-pinned-python-environment) - [How do Chromium and ChromeDriver versions stay compatible?](#how-do-chromium-and-chromedriver-versions-stay-compatible) - [How do you configure scrapy-selenium4 in a Scrapy project?](#how-do-you-configure-scrapy-selenium4-in-a-scrapy-project) - [Which scrapy-selenium4 settings belong in settings.py?](#which-scrapy-selenium4-settings-belong-in-settings-py) - [How do you wait for JavaScript content in Scrapy Selenium?](#how-do-you-wait-for-javascript-content-in-scrapy-selenium) - [How do you request a rendered page with async start()?](#how-do-you-request-a-rendered-page-with-async-start) - [How do you parse the Selenium HTMLResponse with Scrapy selectors?](#how-do-you-parse-the-selenium-htmlresponse-with-scrapy-selectors) - [How do you scroll with Selenium inside a Scrapy callback?](#how-do-you-scroll-with-selenium-inside-a-scrapy-callback) - [How do you click buttons and submit forms with Scrapy Selenium?](#how-do-you-click-buttons-and-submit-forms-with-scrapy-selenium) - [How do you take screenshots and always close Selenium resources?](#how-do-you-take-screenshots-and-always-close-selenium-resources) - [How does a SeleniumRequest return screenshot bytes?](#how-does-a-seleniumrequest-return-screenshot-bytes) - [Who closes the Selenium driver in scrapy-selenium4?](#who-closes-the-selenium-driver-in-scrapy-selenium4) - [What breaks in Scrapy Selenium and how do you debug it?](#what-breaks-in-scrapy-selenium-and-how-do-you-debug-it) - [Can Scrapy Selenium bypass bot protection?](#can-scrapy-selenium-bypass-bot-protection) - [When should Selenium work use a standalone Scrapfly product?](#when-should-selenium-work-use-a-standalone-scrapfly-product) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ python headless-browser ### How To Take Screenshots In Python? Learn how to take Python screenshots through Selenium and Playwright, including common browser tips and tricks for custo... ](https://scrapfly.io/blog/posts/how-to-take-screenshots-in-python) [ python screenshots ### How to Track Web Page Changes with Automated Screenshots There are many different ways to monitor web page changes and one of the most popular techniques is screenshot tracking.... ](https://scrapfly.io/blog/posts/how-to-track-web-page-changes-using-automated-screenshots) [ screenshots ### How to Automate Website Screenshots with Python & JavaScript Learn how to automate Chrome screenshots with Playwright, Selenium, Puppeteer, browser commands, extensions, and APIs fo... ](https://scrapfly.io/blog/posts/how-to-automate-chrome-screenshots) ## Related Questions - [ Q How to save and load cookies in Selenium? ](https://scrapfly.io/blog/answers/how-to-save-and-load-cookies-in-selenium) - [ Q How to take a screenshot with Selenium? ](https://scrapfly.io/blog/answers/how-to-take-screenshot-with-selenium) - [ Q How to scroll to an element in Selenium? ](https://scrapfly.io/blog/answers/scroll-to-element-selenium) - [ Q How to get page source in Selenium? ](https://scrapfly.io/blog/answers/how-to-get-page-source-in-selenium) Run headless browsers at scale, **1,000 free credits** [Start Free](https://scrapfly.io/register)