[Blog](https://scrapfly.io/blog) / [blocking](https://scrapfly.io/blog/tag/blocking) / [How to Bypass AWS WAF when Web Scraping in 2026](https://scrapfly.io/blog/posts/how-to-bypass-aws-waf-when-web-scraping) # How to Bypass AWS WAF when Web Scraping in 2026 by [Ziad Shamndy](https://scrapfly.io/blog/author/ziad) Jul 24, 2026 14 min read [\#blocking](https://scrapfly.io/blog/tag/blocking) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-bypass-aws-waf-when-web-scraping "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-bypass-aws-waf-when-web-scraping&text=How%20to%20Bypass%20AWS%20WAF%20when%20Web%20Scraping%20in%202026 "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-bypass-aws-waf-when-web-scraping "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-bypass-aws-waf-when-web-scraping) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-bypass-aws-waf-when-web-scraping) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-bypass-aws-waf-when-web-scraping) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-bypass-aws-waf-when-web-scraping) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-bypass-aws-waf-when-web-scraping) Your scraper ran fine for months. Then one morning every request returns a 403 - or a strange HTTP 202 carrying an `x-amzn-waf-action: challenge` header - because the target quietly switched on AWS WAF. The page structure has not changed. Your selectors still look correct. The data never arrives. In this guide, we cover how to confirm an AWS WAF block from its exact response signatures, how AWS WAF Bot Control detects scrapers across five distinct layers, and how to get a working bypass using either a DIY browser approach or Scrapfly's managed Anti Scraping Protection. ## Key Takeaways - AWS WAF blocks show as a 403 or HTTP 202 with `x-amzn-waf-action: challenge`. A missing `aws-waf-token` cookie means the challenge was never passed. - Plain `requests` and `curl` cannot run `challenge.js`, so no token is minted and every request stays blocked. - Datacenter IPs fail the IP reputation layer immediately. Residential or mobile IPs are required. - Bot Control has two tiers: Common (self-identifying bots) and Targeted (evasive scrapers that must pass a JavaScript challenge). - Scrapfly ASP handles IP rotation, the token flow, TLS fingerprinting, and CAPTCHAs automatically. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## What is AWS WAF Bot Control? AWS WAF - also known as Amazon WAF - is Amazon's managed web application firewall that inspects HTTP and HTTPS requests at the network edge before they reach the origin server. Bot Control is the managed rule group within AWS WAF that specifically targets scrapers, crawlers, and automated scanners. AWS WAF integrates with CloudFront, Application Load Balancer, and API Gateway. Each deployment uses a Web ACL, which is an ordered list of rules that inspect the request's IP address, headers, cookies, body, and request rate. The Web ACL evaluates rules in priority order, and each [rule action](https://docs.aws.amazon.com/waf/latest/developerguide/waf-rule-action.html) is one of Allow, Block, Count, CAPTCHA, or Challenge. Rate limiting is a rule type (a rate-based rule), not a standalone action. Allow and Block are terminating, while a valid, unexpired CAPTCHA or Challenge token lets AWS WAF keep inspecting the remaining rules. ### AWS WAF Common vs Targeted Inspection Bot Control operates at two inspection levels. Which level a site uses determines how difficult the bypass will be. Common Bot Control identifies bots that openly announce themselves. Named scraping frameworks, outdated user agent strings, and HTTP clients that do not execute JavaScript are all caught at this tier. No deeper interrogation is needed because the client identifies itself. Targeted Bot Control is the harder tier. AWS WAF injects a JavaScript challenge script, runs silent browser environment and timing checks, and only issues the `aws-waf-token` cookie after those checks pass. A scraper that looks like a browser on the surface still fails if it cannot execute the challenge. Most e-commerce, travel, and SaaS sites that actively block scrapers in 2026 use Targeted Bot Control. Passing Common-tier checks is not enough for a reliable bypass against these targets. [AWS WAF Bot Control components](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control-components.html) provides the authoritative component reference from the defender side. The sections below approach the same content from the scraper's perspective. ## How to Identify an AWS WAF Block An AWS WAF block shows one of two primary response signatures: a 403 Forbidden status with no useful body, or an HTTP 202 response carrying the `x-amzn-waf-action: challenge` header alongside a CAPTCHA or JavaScript challenge page. The 202 path means Bot Control wants the client to solve the challenge before proceeding. The 403 path means the request was blocked outright, usually because the IP reputation check failed before the JavaScript challenge even ran. A third variant exists. Some AWS WAF configurations return HTTP 200 with a challenge HTML page rather than the actual content. In that case, check the response body for challenge keywords in addition to the status code. The fastest way to confirm the block is to inspect the response status, headers, and cookies directly: python```python import requests # Confirmed AWS WAF Bot Control target as of June 2026: www.nordstrom.com url = "https://www.nordstrom.com/browse/shoes/womens" response = requests.get(url, headers={ "User-Agent": ( "Mozilla/5.0 (Windows NT 10.0; Win64; x64) " "AppleWebKit/537.36 (KHTML, like Gecko) " "Chrome/125.0.0.0 Safari/537.36" ) }) print("Status:", response.status_code) print("x-amzn-waf-action:", response.headers.get("x-amzn-waf-action", "not present")) print("aws-waf-token cookie:", response.cookies.get("aws-waf-token", "not set")) print("Body snippet:", response.text[:200]) ``` A challenge response prints status `202` and `x-amzn-waf-action: challenge` with no token cookie. An outright block prints `403` with neither header nor cookie present. A successful request returns `200` and sets the `aws-waf-token` cookie in the response. [How to Bypass Anti-Bot Protection in 2026: All 8 Major VendorsIdentify and bypass Cloudflare, DataDome, PerimeterX, Kasada, Akamai, Incapsula, F5, and AWS WAF with Python code examples and success rate data.](https://scrapfly.io/blog/posts/how-to-bypass-anti-bot-protection) ## How Does AWS WAF Detect Web Scrapers? AWS WAF and Bot Control combine several detection signals, evaluated as Web ACL rules in priority order rather than as one fixed scoring gate. This guide groups the signals scrapers hit most into five layers: IP reputation, TLS and HTTP/2 fingerprint, header validation, the JavaScript challenge that mints the `aws-waf-token`, and rate-based rules. Failing any of them can get a session challenged or blocked. The five layers are not evaluated in complete isolation. A low score on IP reputation increases the scrutiny applied to subsequent layers, which makes the challenge harder to pass even with a well-configured browser fingerprint. ### IP Reputation (Datacenter vs Residential) Datacenter IP ranges, including AWS EC2 and Google Cloud addresses, carry a built-in negative trust signal. AWS WAF's managed rule group includes reputation lists that recognize cloud provider ASN ranges and update continuously. Residential IPs are assigned by consumer ISPs to individual households. Mobile IPs are assigned by carrier networks. Both read as human because they are statistically associated with real browsing activity rather than automated scripts. IP reputation is checked first. A datacenter IP can fail the first layer before any other signal is evaluated. [How to Avoid Web Scraper IP Blocking?How IP addresses are used in web scraping blocking. Understanding IP metadata and fingerprinting techniques to avoid web scraper blocks.](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-ip-addresses) ### TLS and HTTP/2 Fingerprinting TLS fingerprinting captures the cipher suites, extensions, and negotiation order a client sends during the TLS handshake. Python `requests` and standard `curl` produce JA3 signatures that are immediately recognizable as non-browser clients. HTTP/2 fingerprinting works at the same connection layer. Real browsers use HTTP/2 by default and produce consistent frame settings and stream priorities. An HTTP/1.1-only client, or an HTTP/2 client with non-browser frame settings, is detectable before any request headers are read. To check what TLS and HTTP/2 fingerprint your current setup produces, see [Scrapfly's JA3 fingerprint tool](https://scrapfly.io/web-scraping-tools/ja3-fingerprint) and the [HTTP/2 fingerprint test page](https://scrapfly.io/web-scraping-tools/http2-fingerprint). [How TLS Fingerprint is Used to Block Web Scrapers?TLS fingeprinting is a popular way to identify web scrapers that not many developers are aware of. What is it and how can we fortify our scrapers to avoid being detected?](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-tls) ### HTTP Header and Request Validation Real browsers send a consistent set of headers in a consistent order: `User-Agent`, `Accept`, `Accept-Language`, `Sec-Fetch-Site`, `Sec-CH-UA`, and `Referer` among others. AWS WAF rule groups check both the presence and the ordering of these headers. A scraper that sends only `User-Agent` and `Accept`, or sends them in an order that no real browser produces, fails header validation even when the individual values look correct. [How Headers Are Used to Block Web Scrapers and How to Fix ItIntroduction to web scraping headers - what do they mean, how to configure them in web scrapers and how to avoid being blocked.](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-headers) ### The JavaScript Challenge and aws-waf-token The JavaScript challenge is the core mechanism behind Targeted Bot Control. When a request cannot pass the earlier layers silently, AWS WAF serves `challenge.js`, which runs environment probes, timing checks, and browser capability tests inside the visitor's JavaScript engine. If the challenge passes, AWS WAF mints an encrypted session token and stores it in the `aws-waf-token` cookie. Subsequent requests from the same session that carry a valid, unexpired token skip the challenge flow entirely. Standard HTTP libraries have no JavaScript engine. The `challenge.js` script cannot run, no token is minted, and every request from the same session is treated as an unchallenged client by Bot Control. ### Rate Limiting and Anomaly Detection AWS WAF rate limits can be keyed by IP address, cookie value, header value, or a custom aggregation key. This means that rotating IP addresses alone does not reset a session-keyed rate limit tied to the `aws-waf-token` cookie value. Even, bursty, or high-frequency request patterns also trigger anomaly scoring. Real browsers follow irregular, non-linear navigation patterns that are statistically difficult to replicate without deliberate timing randomization. Scrapfly #### Need to bypass anti-bot protection? Scrapfly's Anti-Scraping Protection handles Cloudflare, DataDome, and more — automatically. [Try Free →](https://scrapfly.io/register)## How to Bypass AWS WAF The reliable bypass path pairs a real browser that executes `challenge.js` to earn the token with residential or mobile IPs to pass the IP reputation layer. Browser-matched TLS signatures and correct header ordering must be in place before the JavaScript challenge runs. The DIY approach uses a stealth browser automation tool running on a residential proxy. The browser executes the challenge, the `aws-waf-token` is minted, and the scraper captures the cookie value for reuse on subsequent HTTP requests. The code below sketches the capture-then-reuse pattern to show where it becomes fragile at scale: python```python import requests from playwright.sync_api import sync_playwright # Step 1: use a real browser on a residential proxy to pass the JS challenge with sync_playwright() as p: browser = p.chromium.launch( headless=True, proxy={"server": "http://residential-proxy-host:port"} ) context = browser.new_context() page = context.new_page() page.goto("https://www.nordstrom.com/browse/shoes/womens") page.wait_for_timeout(4000) # allow challenge.js to run and mint the aws-waf-token cookies = context.cookies() waf_token = next( (c["value"] for c in cookies if c["name"] == "aws-waf-token"), None ) browser.close() # Step 2: replay the token on faster HTTP requests until it expires if waf_token: session = requests.Session() session.cookies.set("aws-waf-token", waf_token) response = session.get("https://www.nordstrom.com/browse/shoes/womens") print("Status:", response.status_code) ``` This pattern works for a small number of requests but breaks at scale. Token expiry forces frequent browser runs. Detection updates change the challenge mechanism. Profile reuse across sessions can trigger behavioral anomaly flags. The table below maps each bypass approach to the detection layers it actually addresses: | Approach | IP layer | TLS / HTTP/2 | Headers | JS challenge | Rate limits | |---|---|---|---|---|---| | Plain HTTP library | Fails | Fails | Partial | Fails | Partial | | Browser on residential proxy | Passes | Passes | Passes | Passes | Partial | | CAPTCHA solver only | Fails | Fails | Partial | Partial | Fails | | Scrapfly ASP | Passes | Passes | Passes | Passes | Passes | CAPTCHA-solving services address the visual or token-based challenge but leave IP reputation, TLS fingerprinting, and behavioral scoring unaddressed. Combining a solver with a datacenter IP still fails at the first layer. The managed alternative removes all five problems from your scraper code with a single API parameter. ## Bypass AWS WAF with Scrapfly Scrapfly's Web Scraping API with `asp=True` handles IP rotation, the `challenge.js` token flow, TLS and HTTP/2 fingerprinting, header consistency, and CAPTCHA solving automatically. A request that returns 403 or 202 on plain `requests` returns a normal 200 with real page content through Scrapfly ASP. ScrapFly's [Web Scraping API](https://scrapfly.io/web-scraping-api) is a single HTTP endpoint for collecting web data at scale, with a **99.99% success rate** across **130M+ proxies in 190+ countries**. - [Anti-Scraping Protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - automatically defeats Cloudflare, DataDome, PerimeterX, Akamai, and 90+ other bot systems. - [Smart proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) - residential and datacenter pools with country and ASN-level geo-targeting. - [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - render SPAs and dynamic pages through real cloud browsers. - [Browser automation scenarios](https://scrapfly.io/docs/scrape-api/javascript-scenario) - scroll, click, fill forms, and wait for elements without managing a browser fleet. - [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - return pages as HTML, JSON, clean text, or LLM-ready Markdown. - [Session management](https://scrapfly.io/docs/scrape-api/session) - keep cookies, headers, and IPs consistent across multi-step flows. - [Smart caching](https://scrapfly.io/docs/scrape-api/getting-started#api_param_cache) - cache successful responses to cut cost on repeat scraping jobs. - [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Scrapy](https://scrapfly.io/docs/sdk/scrapy), and [no-code integrations](https://scrapfly.io/docs/integration/getting-started) including Make, n8n, Zapier, LangChain, and LlamaIndex. The code below shows the failing baseline and the working ASP bypass side by side: python```python import requests from scrapfly import ScrapflyClient, ScrapeConfig # Failing baseline: plain requests is blocked by AWS WAF Bot Control response = requests.get("https://www.nordstrom.com/browse/shoes/womens") print("Without ASP:", response.status_code) # Typical output: 403 or 202 with x-amzn-waf-action: challenge # Working bypass: Scrapfly ASP handles all five detection layers automatically scrapfly = ScrapflyClient(key="YOUR_API_KEY") result = scrapfly.scrape(ScrapeConfig( url="https://www.nordstrom.com/browse/shoes/womens", asp=True, render_js=True, country="us", )) print("With ASP:", result.upstream_status_code) print("Content length:", len(result.content)) ``` The `asp=True` flag activates Anti Scraping Protection, which manages residential IP selection, the browser challenge flow, and TLS matching behind the scenes. The `render_js=True` parameter ensures JavaScript executes fully. The `country="US"` parameter routes the request through a US residential IP. Scrapfly achieves a 96% success rate against AWS WAF Bot Control, per the [Anti Scraping Protection page](https://scrapfly.io/bypass/aws-waf). Failed requests are not billed; only successful responses count against your usage quota. The API is not Python-only. Scrapfly provides SDKs for Python, TypeScript, Go, and Rust, plus a Scrapy extension. The `asp=True` flag works identically across all SDKs. ## FAQ Is it legal to scrape AWS WAF-protected pages?Scraping publicly visible data is generally lawful in most jurisdictions. Do not access data behind a login, do not damage the server, and do not collect personally identifiable information without a legal basis under applicable privacy law such as GDPR or CCPA. Always review the target site's Terms of Service before scraping. What is the aws-waf-token cookie?The `aws-waf-token` cookie is an encrypted session token that AWS WAF issues after `challenge.js` passes its environment checks. Subsequent requests must carry a valid, unexpired token or Bot Control challenges the session again. The token lifetime varies by site configuration but is typically between a few minutes and a few hours. What does x-amzn-waf-action: challenge mean?The `x-amzn-waf-action: challenge` response header signals that AWS WAF wants the client to complete a browser challenge before the origin server responds. This header is returned with an HTTP 202 status code and a challenge page as the response body. Can I bypass AWS WAF with datacenter proxies?Rarely, and not reliably at scale. Datacenter IPs carry a strong negative trust signal in AWS WAF's IP reputation layer. Even when the JavaScript challenge is solved, the IP reputation layer can block or challenge the session before the challenge runs. Residential or mobile IPs are required for consistent results. Is AWS WAF the same as AWS Shield or CloudFront?No. CloudFront is Amazon's CDN and is a common deployment point for AWS WAF, but CloudFront does not perform bot detection on its own. AWS Shield is Amazon's DDoS protection product. Bot Control is the specific AWS WAF managed rule group that targets scrapers and automated bots. ## Summary AWS WAF Bot Control blocks scrapers across five layers: IP reputation, TLS and HTTP/2 fingerprinting, header validation, the JavaScript challenge and `aws-waf-token` token flow, and rate-based anomaly detection. A bypass that addresses only one or two of these layers will fail the others. To confirm a block is AWS WAF, look for an HTTP 403 or 202 status, check for the `x-amzn-waf-action: challenge` response header, and verify whether the `aws-waf-token` cookie was set by the server. For small-scale jobs, a stealth browser running on a residential proxy can earn and reuse the token. At production scale, maintaining browser pools, proxy rotation, token expiry logic, and fingerprint updates across multiple targets is a significant ongoing engineering cost. Scrapfly ASP removes all five layers from your codebase with a single `asp=True` flag and achieves a 96% success rate against AWS WAF Bot Control. Visit the [AWS WAF bypass page](https://scrapfly.io/bypass/aws-waf) to start a free trial and review the API documentation. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [What is AWS WAF Bot Control?](#what-is-aws-waf-bot-control) - [AWS WAF Common vs Targeted Inspection](#aws-waf-common-vs-targeted-inspection) - [How to Identify an AWS WAF Block](#how-to-identify-an-aws-waf-block) - [How Does AWS WAF Detect Web Scrapers?](#how-does-aws-waf-detect-web-scrapers) - [IP Reputation (Datacenter vs Residential)](#ip-reputation-datacenter-vs-residential) - [TLS and HTTP/2 Fingerprinting](#tls-and-http-2-fingerprinting) - [HTTP Header and Request Validation](#http-header-and-request-validation) - [The JavaScript Challenge and aws-waf-token](#the-javascript-challenge-and-aws-waf-token) - [Rate Limiting and Anomaly Detection](#rate-limiting-and-anomaly-detection) - [How to Bypass AWS WAF](#how-to-bypass-aws-waf) - [Bypass AWS WAF with Scrapfly](#bypass-aws-waf-with-scrapfly) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ blocking ### How to Bypass Imperva Incapsula when Web Scraping in 2026 In this article we'll take a look at a popular anti bot service Imperva Incapsula anti bot WAF. How does it detect web s... ](https://scrapfly.io/blog/posts/how-to-bypass-imperva-incapsula-anti-scraping) [ curl ### How to Use cURL GET Requests Here's everything you need to know about cURL GET requests and some common pitfalls you should avoid. ](https://scrapfly.io/blog/posts/how-to-use-curl-get-requests) [ http blocking ### Akamai Bot Manager: Understanding \_abck Cookies and Sensor Data Deep technical guide to Akamai Bot Manager detection layers, \_abck and ak\_bmsc cookies, sensor data mechanics, JavaScr... ](https://scrapfly.io/blog/posts/akamai-bot-manager-understanding-abck-cookies-and-sensor-data) ## Related Questions - [ Q How to block resources in Selenium and Python? ](https://scrapfly.io/blog/answers/how-to-block-resources-in-selenium) - [ Q Web scraping - what is HTTP 520 status code? ](https://scrapfly.io/blog/answers/520-status-code) - [ Q What are some PhantomJS alternatives for automating browsers? ](https://scrapfly.io/blog/answers/what-are-some-phantomjs-alternatives) Bypass anti-bot protection automatically, **1,000 free credits** [Start Free](https://scrapfly.io/register)