[Answers](https://scrapfly.io/blog)   /  [headless-browser](https://scrapfly.io/blog/tag/headless-browser)   /  [How to use headless browsers with scrapy?](https://scrapfly.io/blog/answers/how-to-use-headless-browsers-with-scrapy)   # How to use headless browsers with scrapy?

 by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) Apr 23, 2023 1 min read [\#headless-browser](https://scrapfly.io/blog/tag/headless-browser) [\#scrapy](https://scrapfly.io/blog/tag/scrapy) 

 [  ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-use-headless-browsers-with-scrapy "Share on LinkedIn")    

 

 

Python offers several libraries for headless browser control like [Web Scraping with Playwright and Python](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python) or [Web Scraping with Selenium and Python](https://scrapfly.io/blog/posts/web-scraping-with-selenium-and-python) but integrating them with scrapy can be difficult.

To use Playwright with scrapy the [scrapy-playwright](https://github.com/scrapy-plugins/scrapy-playwright) community extension can be used. Scrapy-playwright works by creating a new download handler that is powered by Playwright exclusively. To activate it set the `DOWNLOADER_HANDLER` setting:

python```python
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
# and switch to asyncio reactor as playwright is asynchronous
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
```



Then to enable playwright attach `meta={"playwright": True}` parameter to each outgoing `scrapy.Request` object:

python```python
import scrapy

class PlaywrightSpider(scrapy.Spider):
    name = "playwright-spider"

    def start_requests(self):
        yield scrapy.Request("https://httpbin.dev/get", meta={"playwright": True})
        # or POST request
        yield scrapy.FormRequest(
            url="https://httpbin.dev/post",
            formdata={"foo": "bar"},
            meta={"playwright": True}
        )

    def parse(self, response):
        # 'response' contains the page as seen by the browser
        return {"url": response.url}
```



While `scrapy-playwright` doesn't give full control of the web browser it integrates effortlessly with scrapy Spiders and can be an easy solution for scraping dynamic web content using scrapy.

Alternatively, check out [Scrapfly's scrapy SDK](https://scrapfly.io/docs/sdk/scrapy) with the [headless browser](https://scrapfly.io/docs/scrape-api/javascript-rendering) feature which configures scrapy request to go through Scrapfly's managed cloud browsers.



 

    



Scale Your Web Scraping

Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits.

  No credit card required  1,000 free API credits  Anti-bot bypass included 

 [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) 

 Not ready? Get our newsletter instead. 

 

## Explore this Article with AI

 [ ChatGPT ](https://chat.openai.com/?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-use-headless-browsers-with-scrapy) [ Gemini ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-use-headless-browsers-with-scrapy) [ Grok ](https://x.com/i/grok?text=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-use-headless-browsers-with-scrapy) [ Perplexity ](https://www.perplexity.ai/search/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-use-headless-browsers-with-scrapy) [ Claude ](https://claude.ai/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-use-headless-browsers-with-scrapy) 



 ## Related Articles

 [  

 python xpath 

### Web Scraping With Scrapy: The Complete Guide in 2026

Tutorial on web scraping with scrapy and Python through a real world example project. Best practices, extension highligh...

 

 ](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) [  

 python headless-browser 

### How to Scrape Dynamic Websites Using Headless Web Browsers

Introduction to using web automation tools such as Puppeteer, Playwright, Selenium and ScrapFly to render dynamic websit...

 

 ](https://scrapfly.io/blog/posts/scraping-using-browsers) [  

 blocking 

### What is CreepJS Browser Fingerprint and How to Bypass It

In this article, we will explore the inner workings of CreepJS, one of the prominent browser fingerprinting tools and ho...

 

 ](https://scrapfly.io/blog/posts/browser-fingerprinting-with-creepjs) 

  ## Related Questions

- [ Q What are some PhantomJS alternatives for automating browsers? ](https://scrapfly.io/blog/answers/what-are-some-phantomjs-alternatives)
- [ Q Getting started with Puppeteer Stealth ](https://scrapfly.io/blog/answers/how-to-use-puppeteer-stealth-what-does-it-do)
- [ Q How to add headers to every or some scrapy requests? ](https://scrapfly.io/blog/answers/how-to-add-headers-to-every-or-some-scrapy-requests)
- [ Q What are some BeautifulSoup alternatives in Python? ](https://scrapfly.io/blog/answers/what-are-some-beautifulsoup-alternatives)
 
  



   



 Run headless browsers at scale, **1,000 free credits** [Start Free](https://scrapfly.io/register)