What are scrapy middlewares and how to use them?

Scrapy middlewares are scrapy spider extensions that modify outgoing and incoming connections.

Scrapy middlewares are often used to:

• Retrying and filtering requests and reponses.
• Modify outgoing connections with different header or proxy values. e.g. rotating proxies or user agent strings.
• Collecting and tracking connection performance.

Scrapy comes with several default middlewares that retry common exceptions, handle redirects, track cookies and decompresses compressed responses.

For example, here's a middleware that adds a header to each request:

# middlewares.py
class CustomHeaderMiddleware:
    def process_request(self, request, spider):
        request.headers['x-token'] = "123456"

# settings.py
DOWNLOADER_MIDDLEWARES = {
    'your_project_name.middlewares.CustomHeaderMiddleware': 500,
}