[Answers](https://scrapfly.io/blog)   /  [scrapy](https://scrapfly.io/blog/tag/scrapy)   /  [What are scrapy middlewares and how to use them?](https://scrapfly.io/blog/answers/what-are-scrapy-middlewares-and-how-to-use-them)   # What are scrapy middlewares and how to use them?

 by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) May 02, 2023 1 min read [\#scrapy](https://scrapfly.io/blog/tag/scrapy) 

 [  ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them "Share on LinkedIn")    

 

 

Scrapy middlewares are Scrapy spider extensions that modify outgoing and incoming connections. It's a convenient tool to introduce connection logic to scrapy spiders.

For example, scrapy middlewares are often used to:

- Retry and filter requests and reponses based on their content.
- Modify outgoing connections with different header or proxies
- Collecting and tracking connection performance.

Scrapy comes with several default middlewares that perform common tasks such as:

- retry common exceptions
- handle redirects
- track cookies
- decompresses compressed responses

Being able to define custom middlewares is the real power of scrapy middlewares. For example, here's a middleware that adds a header to each request:

python```python
# middlewares.py
class CustomHeaderMiddleware:
    def process_request(self, request, spider):
        request.headers['x-token'] = "123456"

# settings.py
DOWNLOADER_MIDDLEWARES = {
    'your_project_name.middlewares.CustomHeaderMiddleware': 500,
}
```



In this example, we're adding a `x-token` header to each outgoing request. The `process_request` method is called for each outgoing request and can be used to modify the request object.



 

   Table of Contents















 

   Join the Newsletter  Get monthly web scraping insights 

 

  



Scale Your Web Scraping

Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits.

  No credit card required  1,000 free API credits  Anti-bot bypass included 

 [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) 

 Not ready? Get our newsletter instead. 

 

## Explore this Article with AI

 [ ChatGPT ](https://chat.openai.com/?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) [ Gemini ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) [ Grok ](https://x.com/i/grok?text=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) [ Perplexity ](https://www.perplexity.ai/search/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) [ Claude ](https://claude.ai/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fwhat-are-scrapy-middlewares-and-how-to-use-them) 



 ## Related Articles

 [  

 python xpath 

### Web Scraping With Scrapy: The Complete Guide in 2026

Tutorial on web scraping with scrapy and Python through a real world example project. Best practices, extension highligh...

 

 ](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) [  

 http python 

### Guide to Python requests POST method

Discover how to use Python's requests library for POST requests, including JSON, form data, and file uploads, along with...

 

 ](https://scrapfly.io/blog/posts/how-to-python-requests-post) [  

 python headless-browser 

### Scrapy Splash Guide: Scrape Dynamic Websites With Scrapy

Learn about web scraping with Scrapy Splash, which lets Scrapy scrape dynamic web pages. We'll define Splash, cover inst...

 

 ](https://scrapfly.io/blog/posts/web-scraping-with-scrapy-splash) 

  ## Related Questions

- [ Q How to capture background requests and responses in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-capture-xhr-requests-puppeteer)
- [ Q How to use headless browsers with scrapy? ](https://scrapfly.io/blog/answers/how-to-use-headless-browsers-with-scrapy)
- [ Q How to pass data between scrapy callbacks in Scrapy? ](https://scrapfly.io/blog/answers/how-to-pass-data-between-scrapy-callbacks)
- [ Q What are scrapy pipelines and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-pipelines-and-how-to-use-them)
 
  



   



 Scale your web scraping effortlessly, **1,000 free credits** [Start Free](https://scrapfly.io/register)