[Answers](https://scrapfly.io/blog) / [proxies](https://scrapfly.io/blog/tag/proxies) / [How to rotate proxies in scrapy spiders?](https://scrapfly.io/blog/answers/how-to-rotate-proxies-in-scrapy-spiders) # How to rotate proxies in scrapy spiders? by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) Apr 23, 2023 2 min read [\#proxies](https://scrapfly.io/blog/tag/proxies) [\#scrapy](https://scrapfly.io/blog/tag/scrapy) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-rotate-proxies-in-scrapy-spiders "Share on LinkedIn") To rotate proxies in scrapy a request middleware can be used. Request middlewares are scrapy extensions that intercept outgoing requests and incoming responses. So, we can create a middleware that intercepts outgoing requests and sets a random proxy: python```python # middlewares.py import random from scrapy import signals class ProxyRotationMiddleware: def __init__(self, proxies): self.proxies = proxies @classmethod def from_crawler(cls, crawler): """retrieve proxy list from the settings.py PROXIES variable""" proxies = crawler.settings.get('PROXIES', []) if not proxies: raise ValueError('No proxies found in settings. Please provide a list of proxies in the PROXIES setting.') return cls(proxies) def process_request(self, request, spider): """pick random proxy for every request""" proxy = random.choice(self.proxies) request.meta['proxy'] = proxy spider.logger.debug(f'Using proxy: {proxy}') # settings.py MIDDLEWARES = { # ... 'myproject.middlewares.ProxyRotationMiddleware': 750, # ... } PROXIES = [ "http://111.22.22.33:8000", "http://user:password@111.22.22.33:8000", ] ``` This basic proxy rotation middleware will automatically attach a random proxy from proxy pool to each outgoing request. However, random proxies is not always the most efficient way to rotate proxies when proxy pool consists of varying quality proxies. Some proxies can perform better than others and visa versa, so another approach is to consider proxy peformance using weighted randomization: python```python # middlewares.py import random from scrapy import signals class ProxyRotationMiddleware: def __init__(self, proxies): self.proxies = proxies self.proxy_stats = {proxy: {"used": 0, "banned": False} for proxy in proxies} @classmethod def from_crawler(cls, crawler): proxies = crawler.settings.get('PROXIES', []) if not proxies: raise ValueError('No proxies found in settings. Please provide a list of proxies in the PROXIES setting.') return cls(proxies) def process_request(self, request, spider): """attach weighted random proxy to each request""" proxy = self._select_proxy() request.meta['proxy'] = proxy self.proxy_stats[proxy]["used"] += 1 spider.logger.debug(f'Using proxy: {proxy}') def process_response(self, request, response, spider): """inspect every response and record proxy performance""" proxy = request.meta['proxy'] # e.g. if response is 403 or 429 we can mark the proxy as banned if response.status in (403, 429): # Add any other status codes that indicate a ban self.proxy_stats[proxy]["banned"] = True spider.logger.warning(f'Proxy {proxy} is banned, status code: {response.status}') else: self.proxy_stats[proxy]["banned"] = True spider.logger.info(f'Proxy {proxy} has recovered, status code: {response.status}') return response def _select_proxy(self): """select weighted random proxy based on proxy stats""" total_used = sum(stats["used"] for stats in self.proxy_stats.values()) weights = [ # example: calculate weight by use count and whether proxy is considered to be banned: ((total_used + 1) / (stats["used"] + 1)) * (0.1 if stats["banned"] else 1) for stats in self.proxy_stats.values() ] return random.choices(list(self.proxy_stats.keys()), weights=weights, k=1)[0] ``` Proxy rotation can get very complicated when scraping websites that use anti-scraping protection so for scaling up your scrapy scrapers check out Scrapfly API which integrates with scrapy directly through [scrapy SDK](https://scrapfly.io/docs/sdk/scrapy) and is capable of [bypassing anti scraping protections](https://scrapfly.io/docs/scrape-api/anti-scraping-protection)! Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Explore this Article with AI [ ChatGPT ](https://chat.openai.com/?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-rotate-proxies-in-scrapy-spiders) [ Gemini ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-rotate-proxies-in-scrapy-spiders) [ Grok ](https://x.com/i/grok?text=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-rotate-proxies-in-scrapy-spiders) [ Perplexity ](https://www.perplexity.ai/search/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-rotate-proxies-in-scrapy-spiders) [ Claude ](https://claude.ai/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-rotate-proxies-in-scrapy-spiders) ## Related Articles [ python proxies ### How to Rotate Proxies in Web Scraping In this article we explore proxy rotation. How does it affect web scraping success and blocking rates and how can we sma... ](https://scrapfly.io/blog/posts/how-to-rotate-proxies-in-web-scraping) [ python xpath ### Web Scraping With Scrapy: The Complete Guide in 2026 Tutorial on web scraping with scrapy and Python through a real world example project. Best practices, extension highligh... ](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) [ curl ### How to Use cURL GET Requests Here's everything you need to know about cURL GET requests and some common pitfalls you should avoid. ](https://scrapfly.io/blog/posts/how-to-use-curl-get-requests) ## Related Questions - [ Q How to add headers to every or some scrapy requests? ](https://scrapfly.io/blog/answers/how-to-add-headers-to-every-or-some-scrapy-requests) - [ Q How to pass custom parameters to scrapy spiders? ](https://scrapfly.io/blog/answers/how-to-pass-parameters-to-scrapy-spiders-cli) - [ Q How to pass data between scrapy callbacks in Scrapy? ](https://scrapfly.io/blog/answers/how-to-pass-data-between-scrapy-callbacks) - [ Q How to pass data from start\_requests to parse callbacks in scrapy? ](https://scrapfly.io/blog/answers/how-to-pass-data-from-start-request-to-callbacks-scrapy) Premium rotating proxies for scraping, **1,000 free credits** [Start Free](https://scrapfly.io/register)