[Blog](https://scrapfly.io/blog) / [api](https://scrapfly.io/blog/tag/api) / [How to Build a Proxy Rotation API With mitmproxy](https://scrapfly.io/blog/posts/build-a-proxy-api-rotate-proxies-and-save-bandwidth) # How to Build a Proxy Rotation API With mitmproxy by [Ziad Shamndy](https://scrapfly.io/blog/author/ziad) Aug 27, 2026 34 min read [\#api](https://scrapfly.io/blog/tag/api) [\#proxies](https://scrapfly.io/blog/tag/proxies) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbuild-a-proxy-api-rotate-proxies-and-save-bandwidth "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbuild-a-proxy-api-rotate-proxies-and-save-bandwidth&text=How%20to%20Build%20a%20Proxy%20Rotation%20API%20With%20mitmproxy "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbuild-a-proxy-api-rotate-proxies-and-save-bandwidth "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbuild-a-proxy-api-rotate-proxies-and-save-bandwidth) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbuild-a-proxy-api-rotate-proxies-and-save-bandwidth) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbuild-a-proxy-api-rotate-proxies-and-save-bandwidth) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbuild-a-proxy-api-rotate-proxies-and-save-bandwidth) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbuild-a-proxy-api-rotate-proxies-and-save-bandwidth) We ran the tutorial version of this build against current [mitmproxy](https://mitmproxy.org) and it broke twice. `http.HTTPResponse` doesn't exist in 12.2.3, and `flow.live.change_upstream_proxy_server` is gone too. Anything published before that shift teaches a dead API. Requests rotate, sessions keep one upstream, failures get classified, and RFC 9111-safe responses get cached, all behind a `RotationClient` wrapper that returns one result. This guide builds that layer on mitmproxy 12.2.3. [How to Optimize ProxiesLearn how to optimize proxies for speed, anonymity, and cost. Includes comparisons of proxy vs VPN, and tips for developers using Scrapfly.](https://scrapfly.io/blog/posts/how-to-optimize-proxies) ## Key Takeaways - **Split ownership**, so the addon classifies attempts while a client wrapper owns retries. - **Rotate by default**, staying sticky only for logins and multi-step sessions. - **Status isn't proof**, since a `200` can be a challenge page and `407` means bad config. - **Cache conservatively**, with explicit freshness only and nothing served stale. - **Pick your path**, Web Scraping API managed or Proxy Saver for an existing provider. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## How Does a Proxy Rotation API Work? A proxy rotation API is a gateway that picks an upstream route for each independent request or logical session, on its own. A useful one names who owns selection, session affinity, response validation, health, cooldown, retry budget, cache policy, and observability. It does more than call `random.choice()`. The request flows through a fixed boundary. Your application calls a `RotationClient`retry wrapper, which calls the mitmproxy addon, which applies a routing policy and health/cooldown state, then picks an upstream proxy that talks to the target. The classified response and cache decision travel back through that same chain, but your application only ever receives the wrapper's final bounded result. That boundary matters for [proxy types and protocols](https://scrapfly.io/blog/posts/introduction-to-proxies-in-web-scraping) too: whatever pool you route through, the code below treats it as one of three scopes. - **An independent request**, eligible for rotation on every call. - **A sticky unit of work**, bound to one upstream while that upstream stays healthy. - **A cacheable representation**, eligible only under the shared-cache policy covered later in this guide. Upstream identity and browser or application session identity are separate concerns. The proxy layer owns the first. Your application owns the second, and this build never stores cookies or browser state on its behalf. ### How Do Upstream Proxies Move Through Healthy, Cooldown, and Sticky States? Each upstream in the pool carries one of four states, tracked with timestamps and bounded counters instead of recursive retries. | State | Meaning | |---|---| | Healthy | Selectable for new independent requests | | Leased / sticky | Reserved for one named session while it stays healthy | | Cooldown | Temporarily excluded after a classified connect failure | | Disabled | Removed by an operator, or unhealthy past a configured limit | An upstream only reaches cooldown through the failure policy covered in the health-checks section below. Nothing in this build cools a route on a fixed timer inside a live request. With the state model set, the next section installs mitmproxy and gets one request through it before any routing policy exists. ## How Do You Set Up mitmproxy 12.2.3 for Upstream Proxy Rotation? Pin `mitmproxy==12.2.3` and Python `>=3.12`, both checked against [mitmproxy's PyPI listing](https://pypi.org/project/mitmproxy/12.2.3/) on 2026-08-25. This build also needs `httpx` for `RotationClient` and `pytest` for the test suite later, so install all three now. bash```bash pip install mitmproxy==12.2.3 httpx pytest mitmdump --version ``` Next, write a small addon that only logs each request. This confirms mitmproxy is intercepting traffic before any routing logic exists: python```python from mitmproxy import http def request(flow: http.HTTPFlow) -> None: print("Request URL:", flow.request.pretty_url) ``` Run it with `mitmdump` and send one request through it with curl. `httpbin.dev` mirrors back whatever it receives, which makes it a controlled target for this kind of check: bash```bash mitmdump -s log_addon.py ``` txt```txt [14:21:52.365] Loading script log_addon.py [14:21:52.367] HTTP(S) proxy listening at *:8080. ``` That's the addon loaded and listening on port 8080. It won't decrypt HTTPS yet, so the next section covers the certificate step before you send a real request through it. ### How Should mitmproxy Trust HTTPS Traffic? mitmproxy generates its own certificate authority on first run, in `~/.mitmproxy` by default. To inspect HTTPS traffic, your client has to trust that CA. Point curl at the generated certificate file instead of installing it system-wide: bash```bash curl -x http://127.0.0.1:8080 \ --cacert ~/.mitmproxy/mitmproxy-ca-cert.pem \ https://httpbin.dev/anything ``` json```json { "args": {}, "headers": { "Accept": ["*/*"], "Accept-Encoding": ["gzip"], "Host": ["httpbin.dev"], "User-Agent": ["curl/8.7.1"] }, "origin": "41.239.218.221", "url": "https://httpbin.dev/anything", "method": "GET" } ``` The addon's terminal also printed `Request URL: https://httpbin.dev/anything`, confirming mitmproxy decrypted the request before forwarding it. One quirk from running this live: mitmproxy 12.2.3 enforces strict HTTP/2 header validation, and `httpbin.dev` currently ships a `Content-Security-Policy` header with leading whitespace that trips that check. Setting `--set http2=false` sidesteps it by forcing HTTP/1.1. Only trust this CA on devices or environments you control. A trusted mitmproxy CA can decrypt any HTTPS traffic that passes through it, so keep this setup to development or scraping infrastructure you own. [The official certificate guide](https://docs.mitmproxy.org/stable/concepts/certificates/) covers platform-specific installs. With one request confirmed end to end, the addon is ready for actual routing logic. The next section replaces the log line with upstream selection. ## How Do You Rotate Upstream Proxies Without Breaking Sticky Sessions? The obsolete `flow.live.change_upstream_proxy_server` helper is gone in 12.2.3. The [tagged upstream-switch example](https://github.com/mitmproxy/mitmproxy/blob/v12.2.3/examples/contrib/change_upstream_proxy.py) instead assigns a `ServerSpec` tuple to `flow.server_conn.via`. It replaces the server connection first if one is already open for a different address: python```python """Upstream rotation with sticky sessions, using the tagged ServerSpec pattern.""" import random import time from mitmproxy import http from mitmproxy.connection import Server from mitmproxy.net.server_spec import ServerSpec UPSTREAMS = [ ("127.0.0.1", 8091), ("127.0.0.1", 8092), ] SESSION_HEADER = "X-Rotation-Session" SESSION_LEASE_SECONDS = 120 class UpstreamState: def __init__(self): self.active_leases = 0 self.concurrency_ceiling = 4 self.cooldown_until = 0.0 def healthy(self, now): return now >= self.cooldown_until and self.active_leases < self.concurrency_ceiling class SessionLease: def __init__(self, address, expires_at): self.address = address self.expires_at = expires_at class ProxyRotator: def __init__(self): self.upstreams = {addr: UpstreamState() for addr in UPSTREAMS} self.sessions = {} def _candidates(self, now): return [addr for addr, state in self.upstreams.items() if state.healthy(now)] def _pick(self, now): candidates = self._candidates(now) if not candidates: return None least = min(self.upstreams[a].active_leases for a in candidates) return random.choice([a for a in candidates if self.upstreams[a].active_leases == least]) def _route_to(self, flow: http.HTTPFlow, address): via = ServerSpec(("http", address)) is_proxy_change = address != flow.server_conn.via[1] server_connection_already_open = flow.server_conn.timestamp_start is not None if is_proxy_change and server_connection_already_open: flow.server_conn = Server(address=flow.server_conn.address) flow.server_conn.via = via def request(self, flow: http.HTTPFlow) -> None: session_key = flow.request.headers.pop(SESSION_HEADER, None) now = time.monotonic() address = None if session_key: lease = self.sessions.get(session_key) if lease and now < lease.expires_at and self.upstreams[lease.address].healthy(now): address = lease.address if address is None: address = self._pick(now) if address is None: flow.response = http.Response.make( 503, b"", {"X-Rotation-Failure": "pool-exhausted", "X-Rotation-Retryable": "false"} ) return if session_key: self.sessions[session_key] = SessionLease(address, now + SESSION_LEASE_SECONDS) self.upstreams[address].active_leases += 1 flow.metadata["rotation_upstream"] = address self._route_to(flow, address) ``` `_route_to` is the routing pattern itself. It compares the target address against `flow.server_conn.via[1]`, and only replaces the server connection object when the route changes and a connection is already open. Selection logic lives in `_pick` and `_candidates`, kept separate from the mitmproxy hook so it's unit-testable without a running proxy. `UpstreamState.healthy()` is the concurrency lease: an upstream stops being a candidate once its `active_leases` count hits `concurrency_ceiling`, and leases release in both the success and error paths shown later. Set the ceiling from your own provider or project limits, not a number this article supplies. Running this against two local upstream stubs and sending six independent requests split cleanly across both addresses, confirmed by an `X-Rotation-Route` header the addon adds on the way out: txt```txt X-Rotation-Route: 127.0.0.1:8091 X-Rotation-Route: 127.0.0.1:8091 X-Rotation-Route: 127.0.0.1:8092 X-Rotation-Route: 127.0.0.1:8092 X-Rotation-Route: 127.0.0.1:8091 X-Rotation-Route: 127.0.0.1:8091 ``` Four of the six landed on `8091` and two on `8092`, confirming independent requests rotate rather than sticking to one address. Sending four requests with the same `X-Rotation-Session` header kept every one of them on the same upstream, while a different session key picked independently: txt```txt same session, 4 requests: 127.0.0.1:8091 (all four) new session key: 127.0.0.1:8092 ``` That's rotation and stickiness both working against a live mitmproxy 12.2.3 process. For strategies beyond this build, see [proxy rotation strategies](https://scrapfly.io/blog/posts/how-to-rotate-proxies-in-web-scraping). The next two sections cover when each mode applies here, then when a sticky lease should let go. ### When Should a Proxy Rotation API Pick a New Upstream? | Unit of work | State carried | Rotation policy | Release condition | |---|---|---|---| | Independent GET/HEAD fetch | None | Rotate on every request | N/A, stateless | | Login or multi-step form | Cookies, server-side session | Hold one upstream | Session ends or upstream fails | | Paginated crawl tied to server state | Pagination cursor | Hold one upstream | Crawl completes or upstream fails | | Shopping cart / checkout flow | Server-side cart state | Hold one upstream | Checkout completes or upstream fails | Rotation away from an unhealthy upstream only happens through the failure policy in the next section, never on a fixed timer buried inside a live flow. ### How Should a Sticky Session Keep One Proxy Identity? The client decides the session key, not the proxy. `RotationClient`, covered next, accepts a `session_key` argument and sends it as the `X-Rotation-Session` header shown above. The addon maps that key to an upstream lease with an expiry, shown in `SessionLease` in the code above. Cookies and other browser state stay in your application layer. The selector only ever stores an address and an expiry timestamp, nothing about the pages the session visited. When a leased upstream enters cooldown, the next request under that session key finds `self.upstreams[lease.address].healthy(now)` false, falls through to `_pick`, and gets a fresh upstream with a new lease. Running this across multiple hosts means coordinating that lease map somewhere shared, commonly Redis, which is a deployment concern worth naming here but out of scope for this build. With rotation and stickiness working, the addon still treats every response as a plain success. The next section adds the classification that tells `RotationClient` when a retry is worth it. ## How Do mitmproxy Health Checks and Client-Owned Retries Protect a Proxy Pool? Two components split the work here on purpose. The mitmproxy addon classifies each attempt, validates the page, and owns the health and cooldown registry. A separate `RotationClient` wrapper owns the bounded retry loop for the application. Neither one does the other's job. The addon marks its verdict on the response itself, using headers scoped to this tutorial client rather than anything that would leak upstream credentials: python```python COOLDOWN_SECONDS = 20 EXPECTED_MARKER = b'"ok": true' def _classify(self, flow: http.HTTPFlow): status = flow.response.status_code if status == 407: return "proxy-auth-failure", False, "none" if status == 429: return "rate-limited", True, "none" if status >= 500: # A 5xx from the origin is not proof the proxy itself is broken. # Only a real connect failure (see error()) cools the upstream. return "origin-error", True, "none" if status == 200 and EXPECTED_MARKER not in flow.response.content: return "validation-failed", True, "none" if status == 403: return "forbidden", True, "none" return "ok", False, "none" def response(self, flow: http.HTTPFlow) -> None: address = flow.metadata.get("rotation_upstream") if address is None: return state = self.upstreams[address] state.active_leases = max(0, state.active_leases - 1) failure, retryable, health_action = self._classify(flow) now = time.monotonic() if health_action == "cooldown": state.cooldown_until = now + COOLDOWN_SECONDS flow.response.headers["X-Rotation-Route"] = f"{address[0]}:{address[1]}" flow.response.headers["X-Rotation-Failure"] = failure flow.response.headers["X-Rotation-Retryable"] = "true" if retryable else "false" flow.response.headers["X-Rotation-Health-Action"] = health_action def error(self, flow: http.HTTPFlow) -> None: address = flow.metadata.get("rotation_upstream") if address is None: return state = self.upstreams[address] state.active_leases = max(0, state.active_leases - 1) state.cooldown_until = time.monotonic() + COOLDOWN_SECONDS ``` `_classify` treats a `200` with no expected content marker as a failure. That's the challenge-page case: a target can return `200 OK` on a block page as easily as on a real result, so status alone never proves success. A `403` gets marked retryable without cooling the upstream, because the target may be rejecting the whole request identity rather than that specific proxy. Only `error()`, which mitmproxy calls when the connection to the upstream itself fails, ever sets a cooldown. A 5xx that the origin answered with doesn't, on its own, prove the proxy is broken. Pointing the addon at an upstream address nothing is listening on confirmed what `error()` receives. mitmproxy answers the client with its own `502 Bad Gateway` before the addon gets a chance to attach custom headers. `RotationClient` treats a bare `502` as a transport failure it can retry. That matches the cooldown the addon already applied server-side. txt```txt HTTP/1.1 502 Bad Gateway Server: mitmproxy 12.2.3

[Errno 61] Connect call failed ('127.0.0.1', 8099)

``` `RotationClient` is the piece that owns retries. It never replays a live mitmproxy flow, since [clientplayback.py](https://github.com/mitmproxy/mitmproxy/blob/v12.2.3/mitmproxy/addons/clientplayback.py) rejects that outright: python```python def check(self, f: flow.Flow) -> str | None: if f.live or f == self.inflight: return "Can't replay live flow." ``` Instead it issues a fresh request for every attempt and keeps a bounded loop: python```python """Application-facing client. Owns bounded retries; never replays a live mitmproxy flow.""" import time import httpx SAFE_METHODS = {"GET", "HEAD", "OPTIONS"} class ClientResult: def __init__(self, response, attempts, terminal_reason): self.response = response self.attempts = attempts self.terminal_reason = terminal_reason class RotationClient: def __init__(self, proxy_url, max_attempts=3, timeout=5.0): self.proxy_url = proxy_url self.max_attempts = max_attempts self.timeout = timeout def request(self, method, url, session_key=None, **kwargs): headers = dict(kwargs.pop("headers", None) or {}) if session_key: headers["X-Rotation-Session"] = session_key attempts = 0 last_response = None with httpx.Client(proxy=self.proxy_url, timeout=self.timeout) as client: while attempts < self.max_attempts: attempts += 1 try: response = client.request(method, url, headers=headers, **kwargs) except httpx.TransportError as exc: if method not in SAFE_METHODS: return ClientResult(None, attempts, f"transport-error:{exc}") continue last_response = response if response.status_code == 502: if method not in SAFE_METHODS: return ClientResult(response, attempts, "transport-error") continue retryable = response.headers.get("X-Rotation-Retryable") == "true" if not retryable: return ClientResult(response, attempts, response.headers.get("X-Rotation-Failure", "ok")) if method not in SAFE_METHODS: return ClientResult(response, attempts, "non-idempotent-not-retried") if response.status_code == 429 and "Retry-After" in response.headers: time.sleep(min(float(response.headers["Retry-After"]), 5.0)) return ClientResult(last_response, attempts, "retry-budget-exhausted") ``` `max_attempts` includes the first request, and only `GET`, `HEAD`, and `OPTIONS` ever loop back for a second try. A `POST` or `PATCH` that comes back retryable still returns immediately, since retrying a write risks duplicating it. An idempotency key is worth adding at the application level for writes that need retries, but that's outside what this client does by default. Running the full contract live against mitmproxy 12.2.3 produced exactly what the classification promises: txt```txt 503 then 200: attempts=2, terminal=ok, status=200 max_attempts cutoff: attempts=2, terminal=retry-budget-exhausted, status=500 POST, retryable 500: attempts=1, terminal=non-idempotent-not-retried, status=500 407 through a stub: attempts=1, terminal=proxy-auth-failure, status=407 429 with Retry-After: attempts=2, terminal=retry-budget-exhausted, waited before retry ``` Only the final `ClientResult` reaches the caller in every case. The intermediate 503, the intermediate 429 wait, and every classification header stay inside `RotationClient`. ### How Should a Proxy Rotation API Handle 403, 407, 429, and 200 Block Pages? | Observed failure | Likely layer | Retry same upstream | Try another upstream | Health action | Client-facing result | |---|---|---|---|---|---| | Connection refused | Proxy or network | No | Yes, within budget | Cooldown | Retryable | | `407` | Proxy credentials | No | No | None | Terminal | | Timeout | Proxy or target | No | Yes, within budget | None | Retryable | | Upstream `5xx` | Origin or proxy, ambiguous | No | Yes, within budget | None | Retryable | | Target `403` | Request identity | No | Yes, within budget | None | Retryable | | `429` with `Retry-After` | Target rate limit | Yes, after delay | Only if repeated | None | Retryable, delayed | | `429` without `Retry-After` | Target rate limit | No | Yes, within budget | None | Retryable | | `200` challenge or denial page | Request identity or fingerprint | No | Yes, within budget | None | Retryable | | Empty JavaScript shell | Rendering, not this build's scope | No | No | None | Terminal, needs a renderer | | Malformed client request | Application bug | No | No | None | Terminal | Status alone never proves the root cause here. A `403` and a `200` challenge page both point at request identity, not necessarily the proxy, which is why neither one cools an upstream by default. ### Why Does RotationClient Own Bounded Retries? `RotationClient` is the only piece that talks to the application. It issues a fresh request per attempt and withholds every intermediate response from its caller, only ever returning the final `ClientResult`. The mitmproxy addon is the attempt observer and router underneath it. It classifies each attempt, updates route state, and returns metadata in response headers. It never decides whether to retry. Since replaying a live flow isn't possible in mitmproxy 12.2.3, there's no version of this design where the addon could own retries by replaying what it already saw. A fresh request through `RotationClient` is the only mechanism. ### How Do Concurrency Leases Prevent One Proxy From Being Overused? `UpstreamState.healthy()`, shown earlier, is the full lease check: an address stops being a candidate once `active_leases` reaches `concurrency_ceiling`. `request()` increments the lease when it picks an address, and `response()` and `error()`both decrement it, on the success and failure paths alike. python```python def test_concurrency_ceiling_excludes_a_saturated_upstream(): state = UpstreamState() state.concurrency_ceiling = 1 state.active_leases = 1 assert state.healthy(now=0) is False ``` This test lives in `test.py` alongside the rest of the suite and runs as part of the same pytest pass shown later in this article, not as a standalone illustration. Set the ceiling from your provider's connection limits or your own project budget. This article doesn't supply a number, since the right value depends entirely on what the upstream can sustain. With health, retries, and leases in place, the pool still answers every request from the network. The next section adds a cache in front of it, without letting session data leak into a shared entry. ## How Do You Cache Proxy API Responses Without Leaking Session Data? [RFC 9111](https://www.rfc-editor.org/rfc/rfc9111) lets a shared cache do more than this build allows. This implementation deliberately skips heuristic freshness, revalidation, and stale-on-error, keeping only what it can prove is safe to reuse. python```python """Conservative RFC 9111 shared cache: explicit freshness only, no revalidation, no stale-on-error.""" import time from mitmproxy import http HOP_BY_HOP = { "connection", "keep-alive", "proxy-authenticate", "proxy-authorization", "te", "trailer", "transfer-encoding", "upgrade", } IDENTITY_HEADERS = {"authorization", "cookie", "x-proxy-session", "x-session-id", "x-client-id"} def _parse_cache_control(value): directives = {} for part in (value or "").split(","): part = part.strip() if not part: continue if "=" in part: k, v = part.split("=", 1) directives[k.strip().lower()] = v.strip().strip('"') else: directives[part.lower()] = True return directives def _request_bypasses_cache(request_headers): if any(h.lower() in IDENTITY_HEADERS for h in request_headers.keys()): return True req_cc = _parse_cache_control(request_headers.get("Cache-Control", "")) return bool(req_cc.get("no-store")) class ConservativeSharedCache: def __init__(self, max_entries=500, ttl_cap=300): self.store = {} self.max_entries = max_entries self.ttl_cap = ttl_cap self.log = [] def _vary_key(self, request_headers, vary_value): if not vary_value: return () names = [n.strip() for n in vary_value.split(",") if n.strip()] if "*" in names: return None return tuple((n.lower(), request_headers.get(n, "")) for n in sorted(names)) def request(self, flow: http.HTTPFlow) -> None: if flow.request.method != "GET": return if _request_bypasses_cache(flow.request.headers): return req_cc = _parse_cache_control(flow.request.headers.get("Cache-Control", "")) key = flow.request.url if "no-cache" in req_cc or req_cc.get("max-age") == "0": self.log.append(("forced-origin", key)) return entry = self.store.get(key) if entry is None: self.log.append(("miss", key)) return vary_key = self._vary_key(flow.request.headers, entry["vary"]) if vary_key is None or vary_key != entry["vary_key"]: self.log.append(("miss", key)) return now = time.time() if now >= entry["expires_at"]: del self.store[key] self.log.append(("stale-deletion", key)) return self.log.append(("hit", key)) headers = dict(entry["headers"]) headers["Age"] = str(entry["age_at_store"] + int(now - entry["stored_at"])) flow.response = http.Response.make(entry["status"], entry["body"], headers) ``` `_request_bypasses_cache` is shared by both the read and write path, so an identity-bearing or `no-store` request neither reads a cached entry nor lets its own response get stored. `no-cache` or `max-age=0` still forces an origin fetch without deleting whatever's already stored, since a fresh response might overwrite it below. A `Vary` mismatch is treated as a plain miss, and an expired entry gets deleted immediately rather than served. Storage runs the same identity check independently, since a request that forced origin can still populate the cache if the fresh response qualifies: python```python def _eligible(self, flow: http.HTTPFlow): if flow.response.status_code not in (200, 203, 204): return False, "status-not-cacheable" if _request_bypasses_cache(flow.request.headers): return False, "identity-bearing-or-no-store-request" resp_cc = _parse_cache_control(flow.response.headers.get("Cache-Control", "")) if resp_cc.get("no-store") or resp_cc.get("private") or "no-cache" in resp_cc: return False, "response-cache-control" if "Set-Cookie" in flow.response.headers: return False, "set-cookie" return True, None def _freshness(self, headers): cc = _parse_cache_control(headers.get("Cache-Control", "")) try: if "s-maxage" in cc: freshness = int(cc["s-maxage"]) elif "max-age" in cc: freshness = int(cc["max-age"]) else: return None age = int(headers.get("Age", "0")) except ValueError: return None return freshness - age def response(self, flow: http.HTTPFlow) -> None: if flow.request.method != "GET": return key = flow.request.url eligible, reason = self._eligible(flow) if not eligible: self.log.append(("uncacheable", key, reason)) return vary_value = flow.response.headers.get("Vary", "") vary_key = self._vary_key(flow.request.headers, vary_value) if vary_value and vary_key is None: self.log.append(("uncacheable", key, "vary-star")) return ttl = self._freshness(flow.response.headers) if ttl is None or ttl <= 0: self.log.append(("uncacheable", key, "no-explicit-freshness-or-expired")) return if key not in self.store and len(self.store) >= self.max_entries: self.log.append(("uncacheable", key, "cache-full")) return try: age_at_store = int(flow.response.headers.get("Age", "0")) except ValueError: age_at_store = 0 replaced = key in self.store self.store[key] = { "status": flow.response.status_code, "body": flow.response.content, "headers": { k: v for k, v in flow.response.headers.items() if k.lower() not in HOP_BY_HOP and k.lower() != "set-cookie" }, "vary": vary_value, "vary_key": vary_key, "stored_at": time.time(), "expires_at": time.time() + min(ttl, self.ttl_cap), "age_at_store": age_at_store, } self.log.append(("replacement" if replaced else "store", key)) ``` `_freshness` checks `s-maxage` before `max-age`, matching RFC 9111's rule that a shared cache prefers the shared-specific directive. Neither directive present means this cache treats the response as having no explicit freshness. It doesn't fall back to the `Expires` header, which RFC 9111 also treats as an explicit, non-heuristic freshness source, or to a heuristic guess. The origin's own `Age` header gets read at store time and kept as `age_at_store`, then added to resident time on every later hit, so a response that arrives already aged doesn't get reported as fresher than it is. `min(ttl, self.ttl_cap)` means the configured cap can only shorten what the origin allowed, never extend it. A response carrying `Set-Cookie` never gets stored here, even though RFC 9111 itself permits caching a response that also sets a cookie. That's a stricter rule than the spec requires, chosen to keep this shared cache from ever handing one client a response meant to carry another client's session state. Thirteen cache-behavior tests, covering exactly the rules above, ran clean against this implementation: txt```txt test_stores_and_serves_explicit_fresh_get PASSED test_bypasses_identity_bearing_request PASSED test_rejects_set_cookie_response PASSED test_no_explicit_freshness_is_uncacheable PASSED test_s_maxage_overrides_max_age_for_shared_cache PASSED test_age_is_subtracted_from_freshness PASSED test_vary_star_is_never_stored PASSED test_vary_field_is_part_of_the_key PASSED test_expired_entry_is_deleted_not_served_stale PASSED test_request_no_cache_forces_origin PASSED test_identity_bearing_request_never_gets_a_cache_hit PASSED test_no_store_request_response_is_not_stored PASSED test_age_header_accounts_for_origin_reported_age PASSED 13 passed in 0.13s ``` For a broader look at caching tradeoffs beyond this conservative shared-cache contract, see [how caching fits into a scraping pipeline](https://scrapfly.io/blog/posts/how-to-use-cache-in-web-scraping). ### What Belongs in a Proxy API Cache Key? | Input | Why it matters | Include or bypass | |---|---|---| | Method | Only `GET` responses are stored here | Include, always `GET` | | Normalized URL | Base identity of the cached resource | Include | | Each header named in `Vary` | Response can differ per header value | Include the name and value | | `Authorization`, `Cookie`, session headers | RFC 9111 allows a shared cache to reuse these with `public`, `must-revalidate`, or `s-maxage`. This build is stricter and treats them as identity-bearing on both read and write | Bypass, never a key input | | Policy version | Lets a rule change invalidate old entries | Include | `Vary: *` is never stored under this policy. A response carrying it fails to match on every subsequent lookup by definition, so storing it would only waste memory. ### How Should TTL and Eviction Control Proxy Cache Freshness? Origin freshness is mandatory, not assumed. `s-maxage` overrides `max-age` for this shared cache, and no explicit directive at all means the response gets bypassed rather than cached on a guess. `Age` gets subtracted before the configured cap applies, so the cap can only shorten what the origin promised. A request carrying `no-cache` or `max-age=0` always forces an origin fetch, and a response carrying `no-cache` is never stored, since this build has no revalidation path to make reuse safe later. Expired entries get deleted the moment they're found, and the origin gets a fresh request instead. That holds even when the origin then fails, since serving stale data is worse than serving nothing this cache didn't already have on hand. With safe caching in place, one bandwidth cost is left on the table: assets the scrape doesn't need at all. The next section makes stubbing those opt-in. ## When Should a Proxy Rotation API Stub Images and CSS? Stubbing pays off when a job only needs HTML or JSON and the client would otherwise download assets nobody parses. It's a bandwidth decision, not a default, so this build only stubs a route that opts in: python```python """Opt-in image/CSS stubbing. JavaScript always passes through.""" import posixpath from urllib.parse import urlparse from mitmproxy import http STUB_CONTENT_TYPES = { ".png": "image/png", ".jpg": "image/jpeg", ".jpeg": "image/jpeg", ".gif": "image/gif", ".webp": "image/webp", ".css": "text/css", } TRANSPARENT_PIXEL = bytes.fromhex( "89504e470d0a1a0a0000000d49484452000000010000000108060000001f15c4890000" "000b49444154789c6360000200000500017a5eab3f0000000049454e44ae426082" ) ROUTE_HEADER = "X-Rotation-Route-Name" class ResourceStubber: def __init__(self, enabled_routes): self.enabled_routes = set(enabled_routes) self.stubbed_bytes = 0 def request(self, flow: http.HTTPFlow) -> None: route = flow.request.headers.get(ROUTE_HEADER) if route not in self.enabled_routes: return path = urlparse(flow.request.pretty_url).path ext = posixpath.splitext(path)[1].lower() content_type = STUB_CONTENT_TYPES.get(ext) if content_type is None: return body = TRANSPARENT_PIXEL if content_type.startswith("image/") else b"" flow.response = http.Response.make(200, body, {"Content-Type": content_type}) self.stubbed_bytes += len(body) ``` The extension check parses `flow.request.pretty_url` with `urlparse` and `posixpath.splitext`, so a query string like `?v=2&cache=0` after `hero.png` doesn't defeat the match. JavaScript never appears in `STUB_CONTENT_TYPES`, since a script tag can be the thing generating the content the scrape is after. Five tests confirmed the matching logic, including the query-string and case-sensitivity edge cases: txt```txt test_stubs_image_when_route_is_opted_in PASSED test_query_string_does_not_defeat_extension_match PASSED test_mixed_case_extension_still_matches PASSED test_javascript_always_passes_through PASSED test_route_not_opted_in_passes_everything_through PASSED 5 passed in 0.04s ``` `stubbed_bytes` tracks stubbed savings separately from `self.log` in the cache addon, since the two numbers answer different questions: one is bytes never fetched, the other is bytes reused. With rotation, health, caching, and stubbing all in place, the addon is feature-complete. The next section proves the whole thing works together against a live mitmproxy process. Scrapfly #### Scale your web scraping effortlessly Scrapfly handles proxies, browsers, and anti-bot bypass — so you can focus on data. [Try Free →](https://scrapfly.io/register)## How Do You Test a mitmproxy Rotation API? "Working" only means something if it's falsifiable. The suite below runs the finished addon and `RotationClient` against a real `mitmdump`process, two controlled upstream proxy stubs, and one controlled local target, all launched as subprocesses from a pytest fixture: python```python @pytest.fixture(scope="session") def rotation_stack(): procs = [ subprocess.Popen([sys.executable, str(HERE / "target_server.py")]), subprocess.Popen(["mitmdump", "--listen-port", "8091", "-s", str(HERE / "stub_a.py"), "--set", "upstream_cert=false"]), subprocess.Popen(["mitmdump", "--listen-port", "8092", "-s", str(HERE / "stub_b.py"), "--set", "upstream_cert=false"]), ] _wait_for_port(9000) _wait_for_port(8091) _wait_for_port(8092) procs.append(subprocess.Popen([ "mitmdump", "--listen-port", "8080", "--mode", "upstream:http://127.0.0.1:8091", "--set", "connection_strategy=lazy", "--set", "upstream_cert=false", "-s", str(HERE / "rotation_addon.py"), ])) _wait_for_port(8080) time.sleep(0.5) yield "http://127.0.0.1:8080" for p in procs: p.terminate() for p in procs: p.wait(timeout=5) def test_transient_failure_is_retried_and_caller_sees_only_final_result(rotation_stack): client = RotationClient(rotation_stack, max_attempts=3) result = client.request("GET", "http://127.0.0.1:9000/fail-once", session_key="retry-demo") assert result.attempts == 2 assert result.response.status_code == 200 assert result.terminal_reason == "ok" ``` The remaining cases in the same file follow the same fixture: | Test | What it proves | |---|---| | `test_concurrency_ceiling_excludes_a_saturated_upstream` | A lease at its ceiling stops being a healthy candidate | | `test_independent_requests_rotate_across_upstreams` | Eight independent requests land on both upstream addresses | | `test_sticky_session_keeps_one_upstream` | Four requests with one session key stay on one address | | `test_retry_budget_is_bounded` | `max_attempts=2` against an always-failing route stops at two | | `test_non_idempotent_write_is_sent_exactly_once` | A retryable 500 on a `POST` still runs once | | `test_proxy_auth_failure_is_terminal` | A `407` returns immediately with no retry | Running the full file against mitmproxy 12.2.3 and Python 3.14 produced: txt```txt test.py::test_concurrency_ceiling_excludes_a_saturated_upstream PASSED test.py::test_independent_requests_rotate_across_upstreams PASSED test.py::test_sticky_session_keeps_one_upstream PASSED test.py::test_transient_failure_is_retried_and_caller_sees_only_final_result PASSED test.py::test_retry_budget_is_bounded PASSED test.py::test_non_idempotent_write_is_sent_exactly_once PASSED test.py::test_proxy_auth_failure_is_terminal PASSED 7 passed in 2.00s ``` Every assertion here checks accepted-page validation, route diversity, or attempt counts, never raw HTTP status alone, since a `200` on its own already proved unreliable earlier in this build. Rotation, retries, and caching all treat a proxy as invisible once it's healthy. That's deliberate, and it's also the build's biggest limitation: swapping IPs doesn't touch anything else a target can fingerprint. ## Why Does IP Rotation Still Leave TLS, HTTP2, and Browser Fingerprints Exposed? IP rotation changes the exit address a request comes from. It doesn't touch anything else about that request's identity. A target can still see cookies and session continuity, header order and values, the TLS handshake, HTTP/2 settings and frame order, a browser's rendering fingerprint, and timing and interaction patterns. Each of those layers works independently of which IP the request came from. A fresh IP paired with the same inconsistent client identity underneath it can still get blocked. That's the common failure mode. Rotating proxies alone was never the fix for it, since it only ever solved the IP-reputation and rate-limit half of the problem. For the fingerprinting half, see [why rotating IPs alone doesn't solve proxy detection](https://scrapfly.io/blog/posts/how-to-avoid-proxy-detection). The next section steps back from the code and compares three ways to run this in production. ## Should You Build With mitmproxy, Web Scraping API, or Proxy Saver? | Path | What you operate | Rotation and identity | Cache and bandwidth | Rendering and anti-bot | |---|---|---|---|---| | DIY mitmproxy | Upstream pool, policy, state, and code | Custom rotation and leases | Custom safe cache and stubbing | Separate stack | | Web Scraping API | Scrape configuration | Managed pools and named sessions | Response cache, isolated per project | Separate options on the same request | | Proxy Saver | An existing upstream provider or account | Upstream credentials, plus a Rotating Proxy setting for per-request rotation | Shared, redirect, and CORS caches, plus stubbing | Egress fingerprint handling only | [Web Scraping API](https://scrapfly.io/products/web-scraping-api) is a page-fetch API, not a bare proxy pool you route your own traffic through. [Proxy Saver](https://scrapfly.io/products/proxy-saver) is a forward-proxy optimizer, and it needs an existing upstream connection behind it. Enabling its Rotating Proxy setting disables a meaningful share of its connection-reuse optimization, so it's worth turning on only when per-request rotation is the requirement. Keep the two Scrapfly products' caches and session models separate in your head too. Web Scraping API's response cache and Proxy Saver's shared caches solve different problems. Web Scraping API's own cache feature can't be combined with named sessions on the same request, confirmed in [Scrapfly's session docs](https://scrapfly.io/docs/scrape-api/session). Everything this article built is the tradeoff DIY mitmproxy makes explicit. On [r/webscraping](https://www.reddit.com/r/webscraping/comments/oes2zn/rotating_proxies_providers_solutions/), u/artemakerrr described self-managed proxy infrastructure on cloud functions as "going to be a constant arms race that requires works and maintenance." That's not a knock on the approach. It's the actual cost of owning the credentials and the infrastructure yourself, which is exactly what client work sometimes requires. [What Is a Proxy Server?A proxy server is one of those technologies every developer has heard of, but few truly understand beyond the basics of "it hides my IP address." In reality, proxies sit at the heart of modern networking and enable everything from corporate firewalls to the massive data-collection pipelines that...](https://scrapfly.io/blog/posts/what-is-a-proxy-server) ### When Should Web Scraping API Take Over Proxy Rotation Infrastructure? Every request through the Web Scraping API uses a managed proxy pool that rotates, cools down, and excludes underperforming proxies on its own. The documented public pools are datacenter and residential, and `proxy_pool` plus `country` control which one handles a given request. bash```bash curl "https://api.scrapfly.io/scrape?key=YOUR_KEY&url=https://httpbin.dev/anything&asp=true" ``` Named sessions persist cookies and navigation identity, backed by a best-effort sticky proxy, and `retry` defaults to true for network errors and `5xx` responses. Rendering and [Anti-Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) sit on the same request as separate options when a target needs more than IP rotation alone. SDKs exist for [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Scrapy](https://scrapfly.io/docs/sdk/scrapy), Go, and Rust, so this path isn't Python-only. ### When Does Proxy Saver Optimize an Existing Upstream Provider? Proxy Saver needs an existing upstream proxy connection already in place. It sits between your client and that upstream as a stateless forward proxy, forwarding country, city, and session parameters straight through to the provider behind it. bash```bash curl -x "http://proxyId-YOUR_PROXY_ID:YOUR_SCRAPFLY_KEY@proxy-saver.scrapfly.io:3333" https://httpbin.dev/anything ``` Its own docs describe [shared, redirect, and CORS caching](https://scrapfly.io/docs/proxy-saver/optimizations), junk-traffic stripping against a maintained blocklist, opt-out image and CSS stubbing, connection reuse, TLS and TCP optimization, DNS pre-warming, and automatic request recovery. It doesn't retain cookies, browser profiles, or sticky-session state on its own, and per-request rotation needs its Rotating Proxy setting turned on explicitly. Three operating paths now sit on the table. The closing section is about picking one for the job in front of you, not about which one wins in general. ## How Should You Choose a Proxy Rotation Path? Choose DIY mitmproxy when owning the routing policy is worth the maintenance this article walked through, and your team can keep up with state, tests, and upstream contracts over time. If the proxy inventory type itself is still an open question, start with [how to choose a proxy type for web scraping](https://scrapfly.io/blog/posts/best-proxy-providers-for-web-scraping). If that part's already decided, go straight to [residential proxy providers](https://scrapfly.io/blog/posts/top-5-residential-proxy-providers) or [datacenter proxy providers](https://scrapfly.io/blog/posts/the-best-datacenter-proxies). Choose the Web Scraping API when the actual job is fetching pages, and operating a proxy, anti-bot, and rendering stack isn't something the team wants to own. Choose Proxy Saver when an upstream provider is staying in the architecture regardless, and the job is squeezing bandwidth, connection reuse, or egress fingerprint handling out of that existing setup. [How to Choose a Web Unblocker for Web Scraping (and When a Proxy Is Enough)When does a proxy stop being enough for web scraping? A 2026 guide to picking proxies, web unblockers, or scraping browsers without overpaying.](https://scrapfly.io/blog/posts/how-to-choose-the-best-proxy-unblocker) ## Skip the Upstream Pool With Scrapfly's Web Scraping API Everything this build handles by hand, routing policy, health state, retry ownership, and safe caching, is exactly what a managed fetch API takes off your plate. ScrapFly's [Web Scraping API](https://scrapfly.io/products/web-scraping-api) is a single HTTP endpoint for collecting web data at scale, with a **99.99% success rate** across **130M+ proxies in 190+ countries**. - [Anti-Scraping Protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - automatically defeats Cloudflare, DataDome, PerimeterX, Akamai, and 90+ other bot systems. - [Smart proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) - residential and datacenter pools with country and ASN level geo-targeting. - [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - render SPAs and dynamic pages through real cloud browsers. - [Browser automation scenarios](https://scrapfly.io/docs/scrape-api/javascript-scenario) - scroll, click, fill forms, and wait for elements without managing a browser fleet. - [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - return pages as HTML, JSON, clean text, or LLM ready Markdown. - [Session management](https://scrapfly.io/docs/scrape-api/session) - keep cookies, headers, and IPs consistent across multi step flows. - [Smart caching](https://scrapfly.io/docs/scrape-api/getting-started#api_param_cache) - cache successful responses to cut cost on repeat scraping jobs. - [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Scrapy](https://scrapfly.io/docs/sdk/scrapy), and [no-code integrations](https://scrapfly.io/docs/integration/getting-started) including [Make](https://scrapfly.io/integration/make), [n8n](https://scrapfly.io/integration/n8n), [Zapier](https://scrapfly.io/integration/zapier), [LangChain](https://scrapfly.io/integration/langchain), and [LlamaIndex](https://scrapfly.io/integration/llamaindex). Whichever path fits your team, the routing, health, and caching concepts in this build carry over directly to how you configure it. ### Power your scraping with Scrapfly Forget about getting blocked. Scrapfly handles anti-bot bypasses, browser rendering, and proxy rotation so you can focus on the data. [Try for FREE!](https://scrapfly.io/register) ## FAQ What is a proxy rotation API?It's a programmatically controlled forward-proxy service that selects an upstream route per request or session, tracking health, cooldown, and retry state instead of calling `random.choice()` on a fixed list. Is it legal to build and run a proxy rotation API?Running the proxy layer itself is legal. What you use it to scrape, and whether that respects a target's terms of service and applicable law, is a separate question this article doesn't answer. When should I use a sticky session instead of rotating every request?Use a sticky session for anything carrying server-side state across requests, like a login, a paginated crawl tied to a cursor, or a checkout flow. Independent, stateless fetches should rotate on every request. Does IP rotation stop anti-bot detection on its own?No, rotation only changes the exit address. Cookies, header consistency, TLS and HTTP/2 fingerprints, and browser behavior all stay identifiable regardless of which IP a request came from. Can I combine Web Scraping API's cache with named sessions?No. Scrapfly's own documentation states the cache feature can't be used together with a named session on the same request, since the two solve different, incompatible problems. ## Summary A working proxy rotation API needs more than a list of addresses and `random.choice()`. This build split routing, health and cooldown tracking, retry ownership, and caching into separate, testable pieces, each running against a live mitmproxy 12.2.3 process. The addon classifies attempts and updates route state, while a small `RotationClient` wrapper is the only thing that decides whether to try again. That boundary keeps a `POST` from firing twice and every intermediate failure invisible to the calling application. None of this replaces identity work like TLS and browser fingerprinting, and it isn't meant to. For teams that would rather not operate this stack themselves, Scrapfly's Web Scraping API and Proxy Saver cover the managed and BYOP versions of the same routing, health, and caching ideas. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [How Does a Proxy Rotation API Work?](#how-does-a-proxy-rotation-api-work) - [How Do Upstream Proxies Move Through Healthy, Cooldown, and Sticky States?](#how-do-upstream-proxies-move-through-healthy-cooldown-and-sticky-states) - [How Do You Set Up mitmproxy 12.2.3 for Upstream Proxy Rotation?](#how-do-you-set-up-mitmproxy-12-2-3-for-upstream-proxy-rotation) - [How Should mitmproxy Trust HTTPS Traffic?](#how-should-mitmproxy-trust-https-traffic) - [How Do You Rotate Upstream Proxies Without Breaking Sticky Sessions?](#how-do-you-rotate-upstream-proxies-without-breaking-sticky-sessions) - [When Should a Proxy Rotation API Pick a New Upstream?](#when-should-a-proxy-rotation-api-pick-a-new-upstream) - [How Should a Sticky Session Keep One Proxy Identity?](#how-should-a-sticky-session-keep-one-proxy-identity) - [How Do mitmproxy Health Checks and Client-Owned Retries Protect a Proxy Pool?](#how-do-mitmproxy-health-checks-and-client-owned-retries-protect-a-proxy-pool) - [How Should a Proxy Rotation API Handle 403, 407, 429, and 200 Block Pages?](#how-should-a-proxy-rotation-api-handle-403-407-429-and-200-block-pages) - [Why Does RotationClient Own Bounded Retries?](#why-does-rotationclient-own-bounded-retries) - [How Do Concurrency Leases Prevent One Proxy From Being Overused?](#how-do-concurrency-leases-prevent-one-proxy-from-being-overused) - [How Do You Cache Proxy API Responses Without Leaking Session Data?](#how-do-you-cache-proxy-api-responses-without-leaking-session-data) - [What Belongs in a Proxy API Cache Key?](#what-belongs-in-a-proxy-api-cache-key) - [How Should TTL and Eviction Control Proxy Cache Freshness?](#how-should-ttl-and-eviction-control-proxy-cache-freshness) - [When Should a Proxy Rotation API Stub Images and CSS?](#when-should-a-proxy-rotation-api-stub-images-and-css) - [How Do You Test a mitmproxy Rotation API?](#how-do-you-test-a-mitmproxy-rotation-api) - [Why Does IP Rotation Still Leave TLS, HTTP2, and Browser Fingerprints Exposed?](#why-does-ip-rotation-still-leave-tls-http2-and-browser-fingerprints-exposed) - [Should You Build With mitmproxy, Web Scraping API, or Proxy Saver?](#should-you-build-with-mitmproxy-web-scraping-api-or-proxy-saver) - [When Should Web Scraping API Take Over Proxy Rotation Infrastructure?](#when-should-web-scraping-api-take-over-proxy-rotation-infrastructure) - [When Does Proxy Saver Optimize an Existing Upstream Provider?](#when-does-proxy-saver-optimize-an-existing-upstream-provider) - [How Should You Choose a Proxy Rotation Path?](#how-should-you-choose-a-proxy-rotation-path) - [Skip the Upstream Pool With Scrapfly's Web Scraping API](#skip-the-upstream-pool-with-scrapfly-s-web-scraping-api) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ blocking proxies ### How to Avoid Proxy Detection in Web Scraping Learn what gets proxies flagged, how to pick a proxy type, and when to rotate vs hold a sticky session to avoid detectio... ](https://scrapfly.io/blog/posts/how-to-avoid-proxy-detection) [ python proxies ### How to Rotate Proxies in Web Scraping In this article we explore proxy rotation. How does it affect web scraping success and blocking rates and how can we sma... ](https://scrapfly.io/blog/posts/how-to-rotate-proxies-in-web-scraping) [ python tools ### How to Use Cache In Web Scraping for Major Performance Boost Introduction to web scraping caches. How caching can significantly reduce scraping costs and drastically improve perform... ](https://scrapfly.io/blog/posts/how-to-use-cache-in-web-scraping) ## Related Questions - [ Q How to Rotate Proxies in Scrapy ](https://scrapfly.io/blog/answers/scrapy-spiders-proxy-rotation) - [ Q What are private proxies and how are they used in scraping? ](https://scrapfly.io/blog/answers/what-are-private-proxies-compared-to-shared) - [ Q How to add headers to every or some scrapy requests? ](https://scrapfly.io/blog/answers/how-to-add-headers-to-every-or-some-scrapy-requests) - [ Q Scrapy vs BeautifulSoup: Which Should You Use? ](https://scrapfly.io/blog/answers/scrapy-vs-beautifulsoup) Premium rotating proxies for scraping, **1,000 free credits** [Start Free](https://scrapfly.io/register)