[Answers](https://scrapfly.io/blog) / [proxies](https://scrapfly.io/blog/tag/proxies) / [How to Rotate Proxies in Scrapy](https://scrapfly.io/blog/answers/scrapy-spiders-proxy-rotation) # How to Rotate Proxies in Scrapy by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) Sep 29, 2026 2 min read [\#proxies](https://scrapfly.io/blog/tag/proxies) [\#scrapy](https://scrapfly.io/blog/tag/scrapy) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-spiders-proxy-rotation "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-spiders-proxy-rotation&text=How%20to%20Rotate%20Proxies%20in%20Scrapy "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-spiders-proxy-rotation "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-spiders-proxy-rotation) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-spiders-proxy-rotation) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-spiders-proxy-rotation) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-spiders-proxy-rotation) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fscrapy-spiders-proxy-rotation) A proxy list is not a rotation system. Scrapy needs to know whether the address in `request.meta["proxy"]` is one fixed endpoint, a gateway that rotates upstream, or a raw list the spider must health-check itself. If you mark every 403 as a dead proxy, a target-wide block can drain the entire pool while the real problem remains untouched. ## Key Takeaways - Set a proxy for one Scrapy request through `request.meta["proxy"]`; use a downloader middleware when the same rule applies across the spider. - Treat one rotating gateway differently from a raw proxy list. The gateway owns upstream rotation; a raw list needs selection, failure policy, cooldown, and exhaustion handling. - Register proxy assignment under `DOWNLOADER_MIDDLEWARES`, before Scrapy's `HttpProxyMiddleware` at priority 750. - A 403 or 429 is a target block signal, not proof that one proxy endpoint is dead. Use target-specific ban rules and bounded retries. - Verify the returned exit IP and keep credentials out of logs, stats labels, screenshots, and example output. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## Which Scrapy proxy setup should you use? Pick the implementation by the proxy contract you received, not by whether the exits are datacenter or residential. | What you received | Scrapy route | Who selects the exit | Session behavior | Use | |---|---|---|---|---| | One fixed proxy URL | `request.meta["proxy"]` | Nobody rotates it | Fixed endpoint | One request or a fixed route | | One authenticated rotating gateway | Same metadata, one endpoint | Gateway rotates upstream | Rotating or sticky according to the gateway contract | Provider-managed pool | | Several raw proxy URLs | Rotation middleware | Your spider or package | You define affinity | Self-managed pool | - Do not put one upstream-rotating gateway into a client-side raw-list rotator. - Random selection can choose the same proxy twice. It does not guarantee a different IP for every request. - Proxy types and provider selection are covered in the generic [proxy rotation guide](https://scrapfly.io/blog/posts/how-to-rotate-proxies-in-web-scraping). For Scrapy basics, see [web scraping with Scrapy](https://scrapfly.io/blog/posts/web-scraping-with-scrapy). ## How do you set one proxy on a Scrapy request? Use the native path first. Keep the full authenticated proxy URL in an environment variable and never print it. This spider also serves as the exit-IP check used later: python```python import os import scrapy class CheckProxySpider(scrapy.Spider): name = "check_proxy" custom_settings = {"DOWNLOAD_TIMEOUT": 20} async def start(self): proxy_url = os.getenv("DIRECT_PROXY_URL") for sample in range(1, 6): meta = {"download_timeout": 20} if proxy_url: meta["proxy"] = proxy_url yield scrapy.Request( f"http://api.ipify.org/?format=json&sample={sample}", meta=meta, cb_kwargs={"sample": sample}, dont_filter=True, ) def parse(self, response, sample): yield {"sample": sample, "exit_ip": response.json()["ip"]} ``` The URL format is `scheme://username:password@host:port`, where the scheme is normally `http` or `https`. Load it from a secret and run: shell```shell export DIRECT_PROXY_URL="$SCRAPING_PROXY_URL" scrapy runspider check_proxy.py -O exit.json ``` - `HttpProxyMiddleware`, enabled by default, reads the proxy URL from request metadata. - URL-encode usernames or passwords containing `@`, `:`, `/`, or other URL delimiters. - `HTTPAUTH_USER`, `HTTPAUTH_PASS`, and `HTTPAUTH_DOMAIN` (or the `http_user`/`http_pass`/`http_auth_domain` request meta keys) authenticate to the target, not to the proxy. The spider attributes with those names are deprecated since Scrapy 2.17. ## How do you apply one rotating gateway to every Scrapy request? A gateway exposes one proxy URL while changing or pinning the upstream exit according to its own contract. Assign it in a [downloader middleware](https://scrapfly.io/blog/answers/what-are-scrapy-middlewares-and-how-to-use-them) that runs before `HttpProxyMiddleware`. `middlewares.py`: python```python class GatewayProxyMiddleware: def __init__(self, proxy_url): self.proxy_url = proxy_url @classmethod def from_crawler(cls, crawler): return cls(crawler.settings["PROXY_URL"]) def process_request(self, request): if "proxy" not in request.meta: request.meta["proxy"] = self.proxy_url ``` `settings.py`: python```python import os PROXY_URL = os.environ["GATEWAY_PROXY_URL"] DOWNLOADER_MIDDLEWARES = { "myproject.middlewares.GatewayProxyMiddleware": 350, } ``` Downloader `process_request()` methods run in increasing priority order, and the built-in `HttpProxyMiddleware` sits at 750, so 350 assigns the proxy before it is read. The guard preserves per-request decisions: `meta={"proxy": os.environ["OTHER_PROXY_URL"]}` keeps an explicit route, and `meta={"proxy": None}` opts a request out of proxying. When cookies or login state must stay on one exit, use the gateway's documented sticky-session control. One endpoint does not mean one IP, and it does not mean every request rotates. ## How do you rotate a raw proxy list with `scrapy-rotating-proxies`? [`scrapy-rotating-proxies`](https://pypi.org/project/scrapy-rotating-proxies/) is one option for raw lists, not the automatic choice. shell```shell pip install scrapy-rotating-proxies==0.6.2 ``` `settings.py` (basic list rotation and connectivity only): python```python import os ROTATING_PROXY_LIST = [ proxy_url for proxy_url in os.environ["PROXY_URLS"].split(",") if proxy_url ] DOWNLOADER_MIDDLEWARES = { "rotating_proxies.middlewares.RotatingProxyMiddleware": 610, } ``` shell```shell export PROXY_URLS="$PROXY_A_URL,$PROXY_B_URL" ``` The latest PyPI release is 0.6.2 from May 25, 2019, and its published classifiers stop at older Python versions. Release age does not prove incompatibility. In our basic smoke test, version 0.6.2 rotated ten authenticated requests across two local proxy endpoints on Python 3.14 and Scrapy 2.19.0, while logging deprecation warnings about its old `spider` argument. Treat advanced, target-specific behavior as something to test in your own crawl. `BanDetectionMiddleware` is deliberately absent, because a generic page cannot supply a safe ban policy for your target: - The shown configuration only assigns eligible list entries. It does not mark proxies dead or run the package's retry and cooldown lifecycle. - Health tracking needs `BanDetectionMiddleware` at 620 plus a target-specific `ROTATING_PROXY_BAN_POLICY`. The default policy treats most non-200 or empty responses and downloader exceptions as bans (only 200, 301, and 302 are exempt, so a 307, 308, or 404 counts as a ban, and `IgnoreRequest` is also exempt), and the package docs warn that ban detection is site-specific. - With health tracking on, `ROTATING_PROXY_PAGE_RETRY_TIMES` defaults to 5, `ROTATING_PROXY_BACKOFF_BASE` to 300 seconds, and `ROTATING_PROXY_BACKOFF_CAP` to 3600 seconds. - `ROTATING_PROXY_CLOSE_SPIDER=False` (the default) resets and rechecks dead proxies when none are alive; `True` closes the spider. Choose explicitly. - Once `BanDetectionMiddleware` is enabled, the package logs dead and good proxies with their full URL at DEBUG level, so run with `LOG_LEVEL = "INFO"` when credentials are embedded. - Scrapy's concurrency and delay slots become per proxy host for proxied requests (entries that share a hostname but differ by port share one slot), so a larger pool can raise aggregate traffic to the target. ## How should Scrapy classify proxy failures and retries? | Symptom | What it proves | Default action | |---|---|---| | Connection refused by the proxy endpoint | Endpoint did not accept the connection | Cool down that endpoint; retry within a limit | | Proxy CONNECT/TLS failure | Proxy path failed for this attempt | Record the exception; cool down or inspect configuration | | HTTP 407 | Proxy authentication failed | Fix credentials or account policy; do not churn through the pool | | HTTP 403 or 429 from the target | Target rejected or throttled the request | Treat as target-specific; rotate only if the ban policy supports it | | HTTP 5xx from the target | Target/server failure | Let one bounded retry owner handle it; do not kill the proxy | | HTTP 200 with challenge or empty content | Status alone is insufficient | Validate the body with target-specific rules | | Valid target response | Request succeeded | Keep the proxy eligible and reset its failure state | 1. Give each condition one retry owner. `RetryMiddleware` retries 408, 429, 500, 502, 503, 504, 522, and 524 by default. A package ban policy should return `False` or `None` for those so both layers do not retry the same response. 2. Keep 403 and challenge handling target-specific. Scrapy does not retry 403 by default. Add it only when the target evidence and retry budget justify it. 3. Bound attempts and cooldowns. Don't loop over an empty pool with recursion or a `while True` selector, and don't blindly retry non-idempotent requests, since a repeated POST can duplicate an action. Scrapfly #### Scale your web scraping effortlessly Scrapfly handles proxies, browsers, and anti-bot bypass — so you can focus on data. [Try Free →](https://scrapfly.io/register)## How do rotating proxies affect Scrapy sticky sessions and concurrency? Per-request selection can change the IP and break a cookie-bound or logged-in session. A rotating gateway can pin one exit to a documented session key while the endpoint stays constant; the syntax differs per provider. Give each logical session a stable `request.meta["cookiejar"]` ID and map it to the same raw proxy or gateway session key on later requests. `cookiejar` alone preserves cookies, not the IP, so carry both forward and rotate them together at a deliberate boundary. Scrapy concurrency controls request volume; the gateway or pool controls routing. More exits do not justify unlimited traffic, so keep target-level rate limits and watch aggregate volume. ## How do you verify Scrapy proxy rotation without leaking credentials? Run the five-request `check_proxy` spider above. Unique `sample` values plus `dont_filter=True` make Scrapy send all five, and the 20-second timeout bounds the check. It records only the sample number and exit IP. - Fixed proxy: set `DIRECT_PROXY_URL`. The exit should stay stable. - Gateway: leave `DIRECT_PROXY_URL` unset and run inside the project with `GATEWAY_PROXY_URL` set. Exits follow the gateway policy, and a sticky session should keep one exit. - Raw list: leave `DIRECT_PROXY_URL` unset and run inside the project with `PROXY_URLS` set. Requests should spread across the pool, but random choice may repeat. Output order is not request order when concurrency is above one. Never print `request.meta["proxy"]`, the environment variable, or any credential. Seeing the gateway host in a log is not proof of the exit IP. ## Troubleshooting Scrapy proxy rotation | Symptom | Check | |---|---| | Middleware has no effect or proxy auth fails | Confirm `DOWNLOADER_MIDDLEWARES`, the class path, and priority before 750 | | HTTP 407 | Check proxy credentials, URL encoding, and account policy | | Same IP appears repeatedly | Check for random repeats, sticky gateway behavior, or an inactive middleware | | 403/429 continues | Inspect the target response; rotation alone does not prove the pool failed | | Every proxy becomes dead | Narrow the ban policy; do not treat every non-200 as endpoint failure | | Spider stalls with no live proxy | Set an explicit exhaustion policy: pause/recheck, fail fast, or close | | Credentials appear in logs | Remove full proxy URLs and rotate any exposed secret | | Aggregate traffic jumps | Recheck per-proxy concurrency and target-level limits | ## When should you stop managing Scrapy proxy rotation yourself? 1. Managed fetching: [Web Scraping API proxy routing](https://scrapfly.io/docs/scrape-api/proxy) manages a proxy pool that rotates IPs, cools them down, and excludes underperforming proxies, and you pick the pool per request. It fits when proxy selection, retries, rendering, and bot-protection handling are infrastructure rather than the spider's job. 2. Bring your own proxy provider: [Proxy Saver](https://scrapfly.io/docs/proxy-saver/getting-started) is a forward-proxy egress optimizer for an upstream proxy account and can fill the `GATEWAY_PROXY_URL` setting above. Proxy Saver reuses upstream connections, which overrides a rotating gateway's per-request rotation unless you enable its Rotating Proxy setting, and its docs warn that this setting disables a large share of those connection-reuse optimizations. ## FAQ Does `scrapy-rotating-proxies` work with current Scrapy?The latest release is 0.6.2 from May 25, 2019. Basic authenticated list rotation worked in our Scrapy 2.19.0 smoke test, with deprecation warnings, but that is no guarantee for every Python version, ban policy, or target. Pin and test it against your crawler. Why is Scrapy ignoring my proxy middleware?Usually it is registered under `MIDDLEWARES` instead of `DOWNLOADER_MIDDLEWARES`, the import path is wrong, or it runs after `HttpProxyMiddleware` (750), so credentials in the URL are never converted to a `Proxy-Authorization` header and the request fails. Does random rotation guarantee a new IP for every request?No. Random selection can pick the same entry repeatedly, and a gateway can hold one exit for a sticky session. Verify the returned IP instead of inferring it from the endpoint string. Why do I still get 403 or 429 responses after rotating proxies?A new IP does not change every target signal or lift target-level rate limits. Inspect the response body and headers, reduce request rate where appropriate, and use a target-specific failure policy before marking proxies dead. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [Which Scrapy proxy setup should you use?](#which-scrapy-proxy-setup-should-you-use) - [How do you set one proxy on a Scrapy request?](#how-do-you-set-one-proxy-on-a-scrapy-request) - [How do you apply one rotating gateway to every Scrapy request?](#how-do-you-apply-one-rotating-gateway-to-every-scrapy-request) - [How do you rotate a raw proxy list with scrapy-rotating-proxies?](#how-do-you-rotate-a-raw-proxy-list-with-scrapy-rotating-proxies) - [How should Scrapy classify proxy failures and retries?](#how-should-scrapy-classify-proxy-failures-and-retries) - [How do rotating proxies affect Scrapy sticky sessions and concurrency?](#how-do-rotating-proxies-affect-scrapy-sticky-sessions-and-concurrency) - [How do you verify Scrapy proxy rotation without leaking credentials?](#how-do-you-verify-scrapy-proxy-rotation-without-leaking-credentials) - [Troubleshooting Scrapy proxy rotation](#troubleshooting-scrapy-proxy-rotation) - [When should you stop managing Scrapy proxy rotation yourself?](#when-should-you-stop-managing-scrapy-proxy-rotation-yourself) - [FAQ](#faq) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ python proxies ### How to Rotate Proxies in Web Scraping In this article we explore proxy rotation. How does it affect web scraping success and blocking rates and how can we sma... ](https://scrapfly.io/blog/posts/how-to-rotate-proxies-in-web-scraping) [ python crawling ### Guide to List Crawling: Everything You Need to Know Complete list crawling tutorial assess site defenses, bypass anti-bot systems, choose tools (Beautiful Soup, Playwright,... ](https://scrapfly.io/blog/posts/guide-to-list-crawling) [ python scrapeguide ### How to scrape Threads by Meta using Python (2026 Update) Guide how to scrape Threads - new social media network by Meta and Instagram - using Python and popular libraries like P... ](https://scrapfly.io/blog/posts/how-to-scrape-threads) ## Related Questions - [ Q How to pass data from start\_requests to parse callbacks in scrapy? ](https://scrapfly.io/blog/answers/how-to-pass-data-from-start-request-to-callbacks-scrapy) - [ Q What is the difference between IPv4 vs IPv6 in web scraping? ](https://scrapfly.io/blog/answers/ipv4-vs-ipv6-in-web-scraping) - [ Q What are private proxies and how are they used in scraping? ](https://scrapfly.io/blog/answers/what-are-private-proxies-compared-to-shared) - [ Q How to check if element exists in Playwright? ](https://scrapfly.io/blog/answers/how-to-check-for-element-in-playwright) Premium rotating proxies for scraping, **1,000 free credits** [Start Free](https://scrapfly.io/register)