[Answers](https://scrapfly.io/blog)   /  [scrapy](https://scrapfly.io/blog/tag/scrapy)   /  [What are scrapy middlewares and how to use them?](https://scrapfly.io/blog/answers/what-are-scrapy-middlewares-and-how-to-use-them)   # What are scrapy middlewares and how to use them?

 by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) May 02, 2023 1 min read [\#scrapy](https://scrapfly.io/blog/tag/scrapy) 

 [  ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them "Share on LinkedIn")    

 

 

Scrapy middlewares are Scrapy spider extensions that modify outgoing and incoming connections. It's a convenient tool to introduce connection logic to scrapy spiders.

For example, scrapy middlewares are often used to:

- Retry and filter requests and reponses based on their content.
- Modify outgoing connections with different header or proxies
- Collecting and tracking connection performance.

Scrapy comes with several default middlewares that perform common tasks such as:

- retry common exceptions
- handle redirects
- track cookies
- decompresses compressed responses

Being able to define custom middlewares is the real power of scrapy middlewares. For example, here's a middleware that adds a header to each request:

python```python
# middlewares.py
class CustomHeaderMiddleware:
    def process_request(self, request, spider):
        request.headers['x-token'] = "123456"

# settings.py
DOWNLOADER_MIDDLEWARES = {
    'your_project_name.middlewares.CustomHeaderMiddleware': 500,
}
```



In this example, we're adding a `x-token` header to each outgoing request. The `process_request` method is called for each outgoing request and can be used to modify the request object.



 

    



Scale Your Web Scraping

Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits.

  No credit card required  1,000 free API credits  Anti-bot bypass included 

 [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) 

 Not ready? Get our newsletter instead. 

 

## Explore this Article with AI

 [ ChatGPT ](https://chat.openai.com/?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) [ Gemini ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) [ Grok ](https://x.com/i/grok?text=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) [ Perplexity ](https://www.perplexity.ai/search/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) [ Claude ](https://claude.ai/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) 



 ## Related Articles

 [  

 python xpath 

### Web Scraping With Scrapy: The Complete Guide in 2026

Tutorial on web scraping with scrapy and Python through a real world example project. Best practices, extension highligh...

 

 ](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) [  

 python headless-browser 

### Scrapy Splash Guide: Scrape Dynamic Websites With Scrapy

Learn about web scraping with Scrapy Splash, which lets Scrapy scrape dynamic web pages. We'll define Splash, cover inst...

 

 ](https://scrapfly.io/blog/posts/web-scraping-with-scrapy-splash) [  

 http python 

### Guide to Python requests POST method

Discover how to use Python's requests library for POST requests, including JSON, form data, and file uploads, along with...

 

 ](https://scrapfly.io/blog/posts/how-to-python-requests-post) 

  ## Related Questions

- [ Q How to capture background requests and responses in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-capture-xhr-requests-puppeteer)
- [ Q How to use headless browsers with scrapy? ](https://scrapfly.io/blog/answers/how-to-use-headless-browsers-with-scrapy)
- [ Q How to pass data between scrapy callbacks in Scrapy? ](https://scrapfly.io/blog/answers/how-to-pass-data-between-scrapy-callbacks)
- [ Q What are scrapy pipelines and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-pipelines-and-how-to-use-them)
 
  



   



 Scale your web scraping effortlessly, **1,000 free credits** [Start Free](https://scrapfly.io/register)