[Blog](https://scrapfly.io/blog) / [blocking](https://scrapfly.io/blog/tag/blocking) / [How to Avoid Proxy Detection in Web Scraping](https://scrapfly.io/blog/posts/how-to-avoid-proxy-detection) # How to Avoid Proxy Detection in Web Scraping by [Hisham Medhat](https://scrapfly.io/blog/author/hisham) Jun 30, 2026 16 min read [\#blocking](https://scrapfly.io/blog/tag/blocking) [\#proxies](https://scrapfly.io/blog/tag/proxies) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-avoid-proxy-detection "Share on LinkedIn") **Web Scraping API — Proxy Rotation**Rotate proxies from datacenter or residential pools of 130M+ proxies from 120+ countries. [ Learn More ](https://scrapfly.io/products/web-scraping-api#features) [ Docs ](https://scrapfly.io/docs/scrape-api/getting-started#features) You bought clean residential proxies, rotated them on every request, and still hit a wall of 403s and CAPTCHAs. The IP changed, but the blocks kept coming. That's the part most proxy advice skips. A website doesn't block "a proxy"; it blocks traffic that looks inconsistent or low-trust, no matter how fresh the IP is. This guide stays on the proxy decision layer. You'll learn what gets an IP flagged, how to pick a proxy type, and when to rotate versus hold a sticky session. The last piece matters most: it's where proxies stop being the problem and your client fingerprint takes over. [How to Bypass Anti-Bot Protection When Web ScrapingLearn how anti-bot systems detect scrapers and 5 universal bypass techniques including proxy rotation, fingerprinting, and fortified headless browsers.](https://scrapfly.io/blog/posts/how-to-bypass-anti-bot-protection-when-web-scraping) ## Key Takeaways - **Detection scores trust, not IPs:** IP, behavior, and client consistency count together. - **IP reputation comes first:** ASN, subnet, and geolocation set your scrutiny level. - **Match proxy to difficulty:** datacenter for light targets, mobile for the hardest. - **Residential isn't magic:** bad sessions or fingerprint gaps still get it blocked. - **Rotation isn't always safer:** rotate independent requests, stay sticky for flows. - **Fix in layers:** repair IP quality, then session strategy, then fingerprint hardening. - **Offload it:** Scrapfly runs rotation, sessions, and fingerprinting behind one API. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## What Is Proxy Detection in Web Scraping? Proxy detection is how a website decides whether a request comes from a real user or from automated traffic hiding behind a proxy. To avoid it, your traffic needs to look coherent and trustworthy, not arrive from a different IP address alone. Websites evaluate the full picture. They weigh where the IP comes from and how the session behaves over time. They also check whether your headers and client identity match the user the proxy claims to be. A request can clear one of those checks and still fail the other two. This guide works through those layers in order: IP trust signals, proxy type, session strategy, and when to escalate beyond proxies. Start with how sites run these checks. ## How Do Websites Detect and Block Proxies? Websites detect proxies by checking three things: where the IP comes from, how the session behaves, and whether the client identity is consistent. No single signal blocks you. A pile of weak signals pointing the same way does. Think of it as a trust score rather than an on/off switch. Each layer below adds or removes trust, and the request that gets through is the one that looks ordinary across all of them. ### IP Reputation Checks: ASN, Subnet Contamination, and Geolocation The first thing a site checks is the reputation of the IP itself. Every address belongs to an ASN (the network that owns it), and datacenter ASNs are public knowledge. When thousands of requests come from an Amazon or DigitalOcean range, a site can flag the whole block as non-residential. Reputation also travels by neighborhood. If other scrapers burned addresses in the same subnet, your "fresh" IP inherits that bad history before you send a single request. Geolocation adds another check. An IP that resolves to a different country than your stated language, timezone, or currency reads as a mismatch. Connection type matters too, since residential and mobile IPs carry more trust than a known hosting provider. For the full set of address-level signals, see [how IP addresses get scrapers blocked](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-ip-addresses). ### Suspicious Session Behavior: Request Velocity and Repetitive Patterns The second layer watches how the session behaves over time. Real users browse in bursts, pause, and follow links in no fixed order. Scrapers tend to do the opposite. They fire requests at a steady high velocity and hit the same endpoints in the same sequence. They show no continuity where a human would have some. A session might request a hundred product pages in a minute with no search, no scroll, and no idle time. That looks automated regardless of the IP behind it. The tell is often the absence of normal noise rather than any one aggressive action. Predictable, evenly spaced traffic is itself a signal. ### TLS Fingerprinting and Browser Fingerprint Mismatches The third layer checks whether your client identity is consistent. Before a site even reads your headers, it can fingerprint your [TLS handshake](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-tls), often hashed as JA3 or JA4. It then compares that fingerprint against the browser you claim to be. A Python HTTP client announcing a Chrome User-Agent is an immediate contradiction, since the two produce different handshakes. The same goes for browser fingerprint signals that don't line up with the rest of the request. We cover what to do about these mismatches in the [Why You're Still Getting Blocked With Proxies](#why-youre-still-getting-blocked-with-proxies) section below. All three layers point the same way. Proxy detection is a trust problem, not a single IP check, and the more consistent your traffic looks, the harder it is to flag. ## Residential Proxies vs Datacenter Proxies: Which Should You Use? It depends on how aggressively the target detects automation. Datacenter proxies work for lighter targets, while residential or mobile proxies earn their cost when anti-bot systems inspect IP reputation closely. Pick the cheapest tier that clears the target, then move up only when you see blocks. If you're new to the categories, the [complete guide to proxies for web scraping](https://scrapfly.io/blog/posts/introduction-to-proxies-in-web-scraping) covers the fundamentals this section builds on. ### When Datacenter Proxies Are Good Enough Datacenter proxies are the right default for lower-security targets. They're cheap, fast, and available in large pools, which suits high-volume jobs against sites that don't check IP reputation closely. Use them for public data, APIs without strict bot defenses, and internal targets you control. If a site only rate-limits and doesn't run a serious anti-bot product, paying for residential wastes your budget. ### When to Use Residential Proxies Residential proxies fit stricter anti-bot environments. Their IPs come from real consumer connections, so they carry the trust a datacenter range can't fake. Reach for them when a target checks connection type or reputation closely, and when stealth matters more than raw throughput. They're slower and pricier than datacenter IPs, so the tradeoff only pays off once datacenter proxies start getting blocked. ### When to Upgrade to Mobile Proxies Mobile proxies are the top tier for the most aggressive targets, like social platforms and heavily guarded e-commerce. Their IPs originate from real carrier networks shared by millions of users, which makes them the hardest tier to block without hitting real customers. That shared nature is exactly why sites hesitate to ban them outright. They're also the most expensive and slowest option. Treat them as the escalation you reach for when residential proxies still fail, not a default. ### Why Residential Proxies Still Get Blocked Residential proxies reduce risk, but they don't guarantee invisibility. A clean IP still gets flagged when the rest of the request betrays it. Common reasons a good IP fails: - **Low-quality networks:** some residential pools are oversold or already burned, so the reputation is bad on arrival. - **Bad session behavior:** robotic velocity and rigid patterns flag the session no matter how clean the IP is. - **Geo mismatch:** an IP in one country paired with a mismatched language, timezone, or currency reads as fake. - **Incoherent fingerprints:** a residential IP attached to a Python TLS signature still looks automated. - **Rotation that breaks continuity:** switching IPs often while keeping one static browser identity is its own red flag. In browser-based scraping with [Playwright](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python) or [Puppeteer](https://scrapfly.io/blog/posts/puppeteer-stealth-complete-guide), DNS or WebRTC leaks can also expose your real IP behind the proxy. That risk doesn't apply to plain HTTP clients like `requests` or `httpx`. The table below sums up how the three proxy tiers compare. | Proxy type | Speed | Anonymity | Cost | Best for | |---|---|---|---|---| | Datacenter | Fast | Low | Low | Public data, light or no anti-bot defenses | | Residential | Medium | High | High | Stricter anti-bot targets where trust matters | | Mobile | Slow | Highest | Extreme | Social platforms and the most aggressive targets | [Top 5 Residential Proxy Providers for Web ScrapingResidential Proxies are the most commonly used proxies in web scraping, primarily to avoid web scraper blocking, throttling and captchas.](https://scrapfly.io/blog/posts/top-5-residential-proxy-providers) The lesson is to match proxy type to target difficulty. Residential and mobile proxies help, but bad sessions, geo mismatches, or fingerprint gaps still get them blocked, which leads straight into session strategy. Scrapfly #### Need to bypass anti-bot protection? Scrapfly's Anti-Scraping Protection handles Cloudflare, DataDome, and more — automatically. [Try Free →](https://scrapfly.io/register)## Proxy Rotation vs Sticky Sessions: Which Strategy Avoids Detection? Rotate when each request is independent, and use a sticky session when the site expects a continuing user flow. A sticky session keeps the same IP and cookies for a window of time. That's what a login flow or multi-page checkout looks like to the server. Getting this choice wrong is a common reason clean proxies still get caught. The two patterns below cover most jobs, and you can read more on the mechanics in [how to rotate proxies in web scraping](https://scrapfly.io/blog/posts/how-to-rotate-proxies-in-web-scraping). ### When to Rotate Proxies on Every Request Rotate on every request when the requests have nothing to do with each other. Public listings, disconnected fetches, and wide crawls all fit this shape. Each page stands alone, so a fresh IP per request spreads your footprint across the pool and keeps any single address from drawing attention. There's no session for the site to track, so continuity buys you nothing here. ### When to Use Sticky Sessions (Logins, Carts, Multi-Step Flows) Use a sticky session when the work is one connected flow. Authenticated areas, search refinement, JS-heavy flows, and multi-step actions like adding to a cart all expect the same user to persist. If the IP changes mid-flow, the server sees a logged-in account jumping countries between clicks. That's far more suspicious than one steady IP. Typical sticky windows run 10 to 30 minutes depending on the provider. Sessions much longer than that are unusual for real users and can draw suspicion on their own. A small policy function makes the rotate-versus-sticky decision explicit per task: python```python def proxy_policy(task): """Return the session policy that matches a unit of work.""" if task["multi_step"] or task["needs_login"]: # One identity must carry cookies and IP across the whole flow. return {"mode": "sticky", "session": task["name"], "ttl_minutes": 10} # Independent fetch: a fresh IP per request spreads the footprint. return {"mode": "rotate", "session": None} tasks = [ {"name": "listing-crawl", "multi_step": False, "needs_login": False}, {"name": "checkout-flow", "multi_step": True, "needs_login": False}, {"name": "account-page", "multi_step": False, "needs_login": True}, ] for task in tasks: print(f"{task['name']} -> {proxy_policy(task)}") ``` Running it over these tasks shows the split: independent crawls rotate, while anything with a login or multiple steps holds one identity. bash```bash listing-crawl -> {'mode': 'rotate', 'session': None} checkout-flow -> {'mode': 'sticky', 'session': 'checkout-flow', 'ttl_minutes': 10} account-page -> {'mode': 'sticky', 'session': 'account-page', 'ttl_minutes': 10} ``` ### Proxy Rotation Mistakes That Trigger Detection Rotation backfires when it's mechanical instead of matched to the work. The mistakes below are the ones that turn a defense into a tell: - **Rotating too aggressively:** a new IP on every hop of a single flow breaks the continuity a real user would have. - **Predictable intervals:** switching on a fixed timer is as machine-like as never switching at all. - **Destroying continuity mid-flow:** changing IP during a login or checkout reads as account hijacking. - **Retrying through the same bad pattern:** hammering a failed request the same way only confirms the pattern. - **Holding a sticky session for hours:** stretching one IP past the normal 10 to 30 minute window builds suspicious traffic volume over time. So random IP swapping is not always the safer move. When a site expects session continuity, rotation can raise suspicion instead of lowering it. That sets up the next problem: blocks that have nothing to do with your IP. ## Why You're Still Getting Blocked With Proxies Because harder targets check more than your IP. They look at whether your headers, TLS handshake, and browser fingerprint are consistent with a real user sitting on that IP. A spotless residential address paired with an automated client is still easy to catch. This is the layer where proxy quality stops mattering and client identity takes over. Here's what those targets inspect. ### How Mismatched Headers and User-Agents Get Proxies Flagged Mismatched or missing headers betray the connection even when the IP is clean. Real browsers send a consistent, ordered set of headers, and they change in predictable ways across requests. A scraper that sends a bare `User-Agent`, drops `Accept-Language`, or orders headers oddly stands out right away. The IP says "real user," and the headers say "script." For the details of which headers matter and how to align them, see [how headers are used to block scrapers](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-headers). ### TLS Fingerprinting: Why Your Python Client Looks Like a Bot A Python TLS fingerprint paired with a Chrome User-Agent is an instant contradiction. Anti-bot systems hash the TLS handshake into a JA3 or JA4 value before they even read your headers. Python `requests` produces a handshake that no real Chrome ever would. Tools like `curl_cffi`, the Python binding for curl-impersonate, exist to fix exactly this by mimicking a real browser's handshake. Stealth browsers solve it a different way by being real browsers. You can go deeper in our guide to [impersonating browser TLS and HTTP2 fingerprints](https://scrapfly.io/blog/posts/curl-impersonate-scrape-chrome-firefox-tls-http2-fingerprint). ### How Bot-Like Request Timing Triggers Detection Perfectly timed requests with no interaction signals look bot-like on their own. Humans are irregular, and machines are precise, so precision becomes the giveaway. A few timing patterns that flag a session: - **Uniform spacing:** requests fired at near-millisecond-identical intervals. - **Zero dwell variation:** the same time on every page, with no slow reads or quick skips. - **No interaction:** a multi-page flow with no scroll or click events anywhere. Any one of these is a weak signal, but several together build a strong one. The takeaway is to fix the proxy and session layer first, then escalate to fingerprint hardening only when a target still pushes back. [Bypass Proxy Detection with Browser Fingerprint ImpersonationStop proxy blocks with browser fingerprint impersonation using this guide for Playwright, Selenium, curl-impersonate & Scrapfly](https://scrapfly.io/blog/posts/bypass-proxy-detection-with-browser-fingerprint-impersonation) ## How to Diagnose Why Your Proxies Are Getting Blocked Start with the IP layer, then fix session strategy, and only escalate to browser hardening when the simpler fixes aren't enough. Most proxy failures are IP or session problems, so jumping straight to advanced tooling usually wastes time. Work the layers in the order a site checks them. Each step below rules out a cause before you spend effort on the next. ### Fix the IP Layer First Check IP quality before anything else. Confirm the proxy type matches the target, the geography lines up with your stated locale, and the pool isn't a burned or oversold range. If datacenter IPs get blocked instantly, move to residential. If the geo is wrong, target the right country. Most "my proxies don't work" problems end here. ### Fix Session Strategy Next If the IP layer is clean and blocks continue, look at session behavior. Confirm you chose rotation versus sticky sessions correctly for the kind of work, then check your pacing. Add irregular delays, cap velocity, and preserve continuity through multi-step flows. Behavior fixes are cheap and often resolve blocks that look like IP problems. ### Escalate to Browser and Fingerprint Hardening Only When Needed Only reach for fingerprint work after the IP and session layers are sane. If a target still blocks coherent, well-paced traffic from a good IP, the problem is your client identity. A quick symptom-to-layer map keeps you from over-correcting: - **Instant 403 on the first request:** fix the IP layer (type, geo, or reputation). - **Blocks after a burst of requests:** fix session strategy (pacing and rotation choice). - **Blocks on coherent, well-paced traffic from a good IP:** escalate to TLS and browser fingerprint hardening. Working detection layers in this order means you spend effort where the block lives, which is the same principle behind a managed scraping stack. ## How Scrapfly Handles Proxy Detection for You ScrapFly's [Web Scraping API](https://scrapfly.io/web-scraping-api) is a single HTTP endpoint for collecting web data at scale, with a **99.99% success rate** across **130M+ proxies in 190+ countries**. - [Anti-Scraping Protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - automatically defeats Cloudflare, DataDome, PerimeterX, Akamai, and 90+ other bot systems. - [Smart proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) - residential and datacenter pools with country and ASN level geo-targeting. - [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - render SPAs and dynamic pages through real cloud browsers. - [Browser automation scenarios](https://scrapfly.io/docs/scrape-api/javascript-scenario) - scroll, click, fill forms, and wait for elements without managing a browser fleet. - [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - return pages as HTML, JSON, clean text, or LLM ready Markdown. - [Session management](https://scrapfly.io/docs/scrape-api/session) - keep cookies, headers, and IPs consistent across multi step flows. - [Smart caching](https://scrapfly.io/docs/scrape-api/getting-started#api_param_cache) - cache successful responses to cut cost on repeat scraping jobs. - [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Scrapy](https://scrapfly.io/docs/sdk/scrapy), and [no-code integrations](https://scrapfly.io/docs/integration/getting-started) including [Make](https://scrapfly.io/integration/make), [n8n](https://scrapfly.io/integration/n8n), [Zapier](https://scrapfly.io/integration/zapier), [LangChain](https://scrapfly.io/integration/langchain), and [LlamaIndex](https://scrapfly.io/integration/llamaindex). ### Web Scraping API Scrape any website with our powerful API. Anti-bot bypass, JavaScript rendering, and rotating proxies built-in. [Try Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started) ## FAQ Are residential proxies enough to avoid detection?Not by themselves. Residential IPs raise trust, but bad session behavior, geo mismatches, or an inconsistent TLS and browser fingerprint will still get them blocked. Should you rotate proxies on every request?Only when each request is independent, like crawling public listings. For logins, carts, or any multi-step flow, hold a sticky session so the IP and cookies stay consistent. Why am I still getting blocked even with residential proxies?The block usually isn't the IP anymore. Mismatched headers, a Python TLS handshake, or robotic request timing flag the client, so fix those layers after the IP and session layers are clean. ## Summary Avoiding proxy detection comes down to making your traffic look coherent and trustworthy, not endlessly swapping IPs. Sites weigh IP reputation, session behavior, and client consistency together, so a fresh IP attached to robotic timing or a Python fingerprint still gets caught. Choose the proxy tier that matches the target, then match your session strategy to the work. Rotate independent requests, and hold a sticky session when a flow expects continuity. When clean, well-paced traffic from a good IP still gets blocked, that's the signal to escalate to header and fingerprint hardening. Work the layers in order and most blocks resolve before you ever touch advanced tooling. Some teams would rather not build and maintain all of this. A managed service like Scrapfly handles proxy rotation, session management, and fingerprinting behind one endpoint. Self-managed stacks remain a fine choice for teams with the time and setup to run them. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [What Is Proxy Detection in Web Scraping?](#what-is-proxy-detection-in-web-scraping) - [How Do Websites Detect and Block Proxies?](#how-do-websites-detect-and-block-proxies) - [IP Reputation Checks: ASN, Subnet Contamination, and Geolocation](#ip-reputation-checks-asn-subnet-contamination-and-geolocation) - [Suspicious Session Behavior: Request Velocity and Repetitive Patterns](#suspicious-session-behavior-request-velocity-and-repetitive-patterns) - [TLS Fingerprinting and Browser Fingerprint Mismatches](#tls-fingerprinting-and-browser-fingerprint-mismatches) - [Residential Proxies vs Datacenter Proxies: Which Should You Use?](#residential-proxies-vs-datacenter-proxies-which-should-you-use) - [When Datacenter Proxies Are Good Enough](#when-datacenter-proxies-are-good-enough) - [When to Use Residential Proxies](#when-to-use-residential-proxies) - [When to Upgrade to Mobile Proxies](#when-to-upgrade-to-mobile-proxies) - [Why Residential Proxies Still Get Blocked](#why-residential-proxies-still-get-blocked) - [Proxy Rotation vs Sticky Sessions: Which Strategy Avoids Detection?](#proxy-rotation-vs-sticky-sessions-which-strategy-avoids-detection) - [When to Rotate Proxies on Every Request](#when-to-rotate-proxies-on-every-request) - [When to Use Sticky Sessions (Logins, Carts, Multi-Step Flows)](#when-to-use-sticky-sessions-logins-carts-multi-step-flows) - [Proxy Rotation Mistakes That Trigger Detection](#proxy-rotation-mistakes-that-trigger-detection) - [Why You're Still Getting Blocked With Proxies](#why-you-re-still-getting-blocked-with-proxies) - [How Mismatched Headers and User-Agents Get Proxies Flagged](#how-mismatched-headers-and-user-agents-get-proxies-flagged) - [TLS Fingerprinting: Why Your Python Client Looks Like a Bot](#tls-fingerprinting-why-your-python-client-looks-like-a-bot) - [How Bot-Like Request Timing Triggers Detection](#how-bot-like-request-timing-triggers-detection) - [How to Diagnose Why Your Proxies Are Getting Blocked](#how-to-diagnose-why-your-proxies-are-getting-blocked) - [Fix the IP Layer First](#fix-the-ip-layer-first) - [Fix Session Strategy Next](#fix-session-strategy-next) - [Escalate to Browser and Fingerprint Hardening Only When Needed](#escalate-to-browser-and-fingerprint-hardening-only-when-needed) - [How Scrapfly Handles Proxy Detection for You](#how-scrapfly-handles-proxy-detection-for-you) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Explore this Article with AI [ ChatGPT ](https://chat.openai.com/?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-avoid-proxy-detection) [ Gemini ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-avoid-proxy-detection) [ Grok ](https://x.com/i/grok?text=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-avoid-proxy-detection) [ Perplexity ](https://www.perplexity.ai/search/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-avoid-proxy-detection) [ Claude ](https://claude.ai/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-avoid-proxy-detection) ## Related Articles [ python proxies ### How to Rotate Proxies in Web Scraping In this article we explore proxy rotation. How does it affect web scraping success and blocking rates and how can we sma... ](https://scrapfly.io/blog/posts/how-to-rotate-proxies-in-web-scraping) [ proxies ### The Best Datacenter Proxies in 2026: A Complete Guide Datacenter proxies are a top choice for web scraping, automation, and online anonymity thanks to their speed, low cost, ... ](https://scrapfly.io/blog/posts/the-best-datacenter-proxies) [ http python ### How to Effectively Use User Agents for Web Scraping In this article, we’ll take a look at the User-Agent header, what it is and how to use it in web scraping. We'll also ge... ](https://scrapfly.io/blog/posts/user-agent-header-in-web-scraping) ## Related Questions - [ Q How to rotate proxies in scrapy spiders? ](https://scrapfly.io/blog/answers/how-to-rotate-proxies-in-scrapy-spiders) - [ Q Mobile vs Residential Proxies - which to choose for scraping? ](https://scrapfly.io/blog/answers/mobile-vs-residential-proxies-whats-the-difference) - [ Q How to save and load cookies in Python requests? ](https://scrapfly.io/blog/answers/save-and-load-cookies-in-requests-python) - [ Q How to get file type of an URL in Python? ](https://scrapfly.io/blog/answers/how-to-get-url-filetype-in-python) Bypass anti-bot protection automatically, **1,000 free credits** [Start Free](https://scrapfly.io/register)