[Answers](https://scrapfly.io/blog)   /  [scrapy](https://scrapfly.io/blog/tag/scrapy)   /  [What are scrapy middlewares and how to use them?](https://scrapfly.io/blog/answers/what-are-scrapy-middlewares-and-how-to-use-them)   # What are scrapy middlewares and how to use them?

 by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) May 02, 2023 1 min read [\#scrapy](https://scrapfly.io/blog/tag/scrapy) 

 [  ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them "Share on LinkedIn") [  ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them&text=What%20are%20scrapy%20middlewares%20and%20how%20to%20use%20them%3F "Share on X") [  ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them "Share on Facebook")    

 

 

Summarize this article with

 [  ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) [  ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) [  ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) [  ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) [  ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) 



Scrapy middlewares are Scrapy spider extensions that modify outgoing and incoming connections. It's a convenient tool to introduce connection logic to scrapy spiders.

For example, scrapy middlewares are often used to:

- Retry and filter requests and reponses based on their content.
- Modify outgoing connections with different header or proxies
- Collecting and tracking connection performance.

Scrapy comes with several default middlewares that perform common tasks such as:

- retry common exceptions
- handle redirects
- track cookies
- decompresses compressed responses

Being able to define custom middlewares is the real power of scrapy middlewares. For example, here's a middleware that adds a header to each request:

python```python
# middlewares.py
class CustomHeaderMiddleware:
    def process_request(self, request, spider):
        request.headers['x-token'] = "123456"

# settings.py
DOWNLOADER_MIDDLEWARES = {
    'your_project_name.middlewares.CustomHeaderMiddleware': 500,
}
```



In this example, we're adding a `x-token` header to each outgoing request. The `process_request` method is called for each outgoing request and can be used to modify the request object.



 

   [  Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io)   Join the Newsletter  Get monthly web scraping insights 

 

  



Scale Your Web Scraping

Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits.

  No credit card required  1,000 free API credits  Anti-bot bypass included 

 [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) 

 Not ready? Get our newsletter instead. 

 

 ## Related Articles

 [  

 python xpath 

### Web Scraping With Scrapy: The Complete Guide in 2026

Tutorial on web scraping with scrapy and Python through a real world example project. Best practices, extension highligh...

 

 ](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) [  

 http python 

### Guide to Python requests POST method

Discover how to use Python's requests library for POST requests, including JSON, form data, and file uploads, along with...

 

 ](https://scrapfly.io/blog/posts/how-to-python-requests-post) [     

 python ai 

### How to Scrape ChatGPT Responses in 2026

Scrape ChatGPT responses with Python: when to pick the OpenAI API, how to handle Cloudflare and streaming responses, and...

 

 ](https://scrapfly.io/blog/posts/how-to-scrape-chatgpt) 

  ## Related Questions

- [ Q How to capture background requests and responses in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-capture-xhr-requests-puppeteer)
- [ Q How to use headless browsers with scrapy? ](https://scrapfly.io/blog/answers/how-to-use-headless-browsers-with-scrapy)
- [ Q What are scrapy pipelines and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-pipelines-and-how-to-use-them)
- [ Q What are scrapy Item and ItemLoader objects and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-items-and-itemloaders)
 
  



   



 Scale your web scraping effortlessly, **1,000 free credits** [Start Free](https://scrapfly.io/register)