[Blog](https://scrapfly.io/blog) / [python](https://scrapfly.io/blog/tag/python) / [Web Scraping Best Practices: 11 Rules for Reliable and Respectful Scraping in 2026](https://scrapfly.io/blog/posts/web-scraping-best-practices) # Web Scraping Best Practices: 11 Rules for Reliable and Respectful Scraping in 2026 by [Mohab Yousry](https://scrapfly.io/blog/author/mohab-yousry-9396552a) Jul 21, 2026 22 min read [\#python](https://scrapfly.io/blog/tag/python) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [\#web-scraping](https://scrapfly.io/blog/tag/web-scraping) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-best-practices "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-best-practices&text=Web%20Scraping%20Best%20Practices%3A%2011%20Rules%20for%20Reliable%20and%20Respectful%20Scraping%20in%202026 "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-best-practices "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-best-practices) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-best-practices) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-best-practices) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-best-practices) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-best-practices) Most scrapers do not die from anti-bot systems. They die quietly, from a renamed CSS class, an IP ban triggered by hundreds of concurrent requests fired from a single script, or a schema change that fills a database with nulls before anyone notices. None of that requires a sophisticated adversary. It just requires a scraper that was never built to survive contact with a production website. The 11 practices below are what turns a script into a pipeline. They are sequenced in the order a real scraping project actually meets them, starting with whether you should send a request at all and ending with how you find out, months later, that something quietly broke. ## Key Takeaways - **Check for an official API or bulk export first.** The best scraping request is the one you never send, and documented access is often faster to build against than HTML. - **Respect robots.txt, Crawl-delay, and the site's terms.** Reading the file takes one request, and following it is the professional standard even though it is not a technical enforcement mechanism. - **Rate limit requests and prefer off-peak hours.** Cap concurrency per domain and pace requests so your traffic never looks like an overload event. - **Prefer hidden APIs and embedded JSON over HTML parsing.** Structured responses from background requests or embedded JSON blocks are more stable and cheaper to parse than deep CSS selectors. - **Use headless browsers only when JavaScript rendering demands it.** Try plain HTTP first, and reserve browser automation for pages that return an empty shell without it. - **Rotate proxies, User-Agents, and fingerprints responsibly.** Distribute identity signals to avoid false-positive blocks, not to keep hitting a site that has already refused you. - **Handle errors with retries and exponential backoff.** Classify errors as retryable or terminal, retry with growing waits plus jitter, and cap the attempt count. - **Treat 403, 429, and CAPTCHA responses as signals, not challenges.** A block is often feedback about your request pattern, and the professional response is to slow down, not escalate. - **Cache responses and deduplicate requests.** Reuse unchanged content with conditional requests and a visited-URL set instead of fetching the same page twice. - **Validate data at extraction and store it with provenance.** Schema-check every record as it is parsed and record the source URL, timestamp, and scraper version alongside it. - **Monitor your scrapers and alert on breakage.** Track success rate, records per run, and freshness, because a scraper can return HTTP 200 while extracting nothing. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## Why Do Most Web Scrapers Fail Without Best Practices? Most scrapers fail not because scraping is technically hard, but because they ignore how production websites behave over time. A script that works perfectly against today's HTML has made no allowance for tomorrow's deploy, and that gap is where most scraping projects quietly die. The failure modes are predictable, and the same handful of them show up in almost every postmortem: - **Brittle selectors break on the next deploy.** Scrapers keyed to a specific CSS class or a deep XPath return empty or wrong fields the moment the DOM changes with no error to signal that anything went wrong. - **Self-inflicted IP bans.** Async HTTP clients make it trivial to fire hundreds of concurrent requests from a single machine, and a site can answer that pattern with rate limits or an outright block of your traffic. - **JavaScript-rendered pages return empty shells.** A plain HTTP client only gets the skeleton HTML when the actual data loads client-side, so the response looks successful while carrying none of the content you wanted. - **Silent data corruption.** Unvalidated parsing fills storage with nulls and misaligned fields that only surface weeks later, when someone downstream asks why the analysis looks wrong. - **Compliance exposure.** Scraping login-gated pages or personal data without checking the applicable terms and regulations creates legal risk that no proxy pool or retry logic can fix. For the traffic-pattern side of this list, detection and blocking mechanics get the full treatment in our dedicated guide. [5 Tools to Scrape Without Blocking and How it All WorksTutorial on how to avoid web scraper blocking. What is javascript and TLS (JA3) fingerprinting and what role request headers play in blocking.](https://scrapfly.io/blog/posts/how-to-scrape-without-getting-blocked-tutorial) Every practice that follows removes one of these failure classes before it reaches production. ## 1. Check for an Official API or Bulk Export Before Scraping The best scraping request is the one you never send. Official APIs and bulk data exports provide documented, structured access, and when they cover the fields you need, they are almost always less work to maintain than a scraper. Start by looking in three places: - Documented REST or GraphQL APIs listed on the site's developer pages - Bulk CSV or JSON downloads on government and open-data portals - A scheduled export feature buried in an account settings page A ten-minute search here can save weeks of selector maintenance later. Be honest about the tradeoffs, though: - Official APIs are often rate limited, paywalled, or missing fields that only exist in the rendered page - Scraping remains the right fallback when structured access does not exist or does not cover what you actually need - The common failure runs the other way too: a team keeps a scraper alive for months to collect data the site already publishes as a scheduled export The diagram shows the full acquisition decision path used throughout this guide. Try each tier in order and only move to the next one when the current tier cannot deliver the data.If there is no official API, there is often an unofficial one running behind the page, which is exactly what the fourth rule on this list covers. ## 2. Respect robots.txt, Crawl-Delay, and the Site's Terms robots.txt is the site owner's machine-readable statement of what automated traffic may access and how fast, and reading it costs exactly one request. Following it is the professional standard for anyone running a scraper at scale. The file lives at the domain root. `User-agent`, `Allow`, and `Disallow` are the core fields defined by RFC 9309, the Robots Exclusion Protocol standardized in 2022. `Crawl-delay` is a widely supported but non-standard extension. Some parsers and sites honor it, and Google explicitly does not, a detail worth checking in [Google's own documentation on interpreting robots.txt](https://developers.google.com/search/docs/crawling/docs/robots-txt/robots-txt-spec) rather than re-deriving it from scratch. A [Duke University study](https://arxiv.org/abs/2505.21733) published May 27, 2025 analyzed traffic from 130 self-declared bots plus a large volume of anonymous traffic over 40 days. Compliance dropped as robots.txt directives got stricter, with AI search crawlers rarely checking the file at all while SEO crawlers stayed the most compliant. [RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309) itself is not a legal enforcement mechanism. robots.txt functions as a norm that depends entirely on the crawler choosing to comply, not a technical gate a server enforces on its own. That is exactly why following it still matters. Honoring `Disallow` and `Crawl-delay` avoids intentionally adding load to paths the owner marked as off-limits, and it makes your access expectations explicit before anyone has to ask. In practice that means parsing the file programmatically, honoring `Crawl-delay` when it is present, and skimming the terms of service for explicit scraping or data-reuse clauses, especially before scraping anything behind a login. The common failure is ignoring a `Disallow` on a high-value path and then losing access entirely when the owner blocks your traffic or your whole ASN in response. python```python import urllib.robotparser rp = urllib.robotparser.RobotFileParser() rp.set_url("https://web-scraping.dev/robots.txt") rp.read() print(rp.can_fetch("*", "https://web-scraping.dev/products")) # True print(rp.can_fetch("*", "https://web-scraping.dev/robots-disallowed")) # False print(rp.crawl_delay("*")) # 2 ``` Managed crawl tooling can enforce this automatically. Scrapfly's [Crawler API](https://scrapfly.io/products/crawler-api) respects robots.txt by default. [How to Crawl the Web with PythonIntroduction to web crawling with Python. What is web crawling? How it differs from web scraping? And a deep dive into code, building our own crawler and an example project crawling Shopify-powered websites.](https://scrapfly.io/blog/posts/crawling-with-python) Following robots.txt is only the first half of staying inside a site's stated limits. The next rule covers how fast to request the pages it does allow. ## 3. Rate Limit Requests and Scrape During Off-Peak Hours Cap your request rate so the site is never treated like it needs to absorb a sudden spike. Excessive request volume can trigger rate limits or outright blocks on the server side, and it degrades the experience for the site's real users at the same time. Use specific numbers instead of vague "be gentle" advice: - A ten-second delay between requests to the same domain is a conservative starting point, following the target's own `Crawl-delay` interval when robots.txt provides one, a baseline the University of Pittsburgh's guide also recommends - Cap concurrent requests per domain - Schedule the heaviest jobs for the target's local off-peak hours rather than the middle of its business day Treat all of these as starting points to tune against the specific site, not as universal limits that apply everywhere. Pacing exists to avoid a constant, predictable request cadence that concentrates load on one endpoint. Jitter helps spread that load out, and it should be used for that reason rather than as an attempt to disguise a scraper as a human. At multi-domain scale, per-domain budgets matter far more than a single global rate, since one polite scraper per site beats one shared rate applied across every target at once. A common failure is an async scraper with no concurrency limit. It fires hundreds of requests at once, triggers a block, and the team blames the anti-bot vendor instead of the self-inflicted spike. To understand more about rate limiting, check our articles: [What is Rate Limiting? Everything You Need to KnowDiscover what rate limiting is, why it matters, how it works, and how developers can implement it to build stable, scalable applications.](https://scrapfly.io/blog/posts/what-is-rate-limiting-everything-you-need-to-know) [How to Rate Limit Async Requests in PythonQuick tutorial on how to limit asynchronous python connections when web scraping. This can reduce and balance out web scraping speed to avoid scraping pages too fast and blocking.](https://scrapfly.io/blog/posts/how-to-rate-limit-asynchronous-python-requests) ## 4. Prefer Hidden APIs and Embedded JSON Over HTML Parsing Before writing a single CSS selector, open DevTools. Many modern sites expose their data through background API calls or embed it directly in the page, and pulling from either source reduces parsing overhead and gives you structured fields instead of text you have to pull out of markup. Two places to look: - The Network tab shows XHR and fetch calls to internal REST or GraphQL endpoints that the page itself uses to load content - The page source often contains embedded state, like JSON-LD blocks meant for search engines or framework payloads such as `__NEXT_DATA__` Either one usually contains more complete data than what actually renders on screen. This is a best practice, not a trick. A structured response is cheaper to parse and transfer than HTML, though an internal endpoint can still change without warning. HTML parsing is still right for server-rendered pages with no data layer, so favor stable attributes over deep positional paths when you write those selectors. The common failure is maintaining dozens of fragile selectors for a page whose data sits, complete, in one embedded JSON object the whole time. python```python import json import httpx from bs4 import BeautifulSoup response = httpx.get("https://web-scraping.dev/product/1") soup = BeautifulSoup(response.text, "html.parser") script = soup.find("script", type="application/ld+json") product = json.loads(script.string) print(product["name"]) print(product["offers"]["lowPrice"], product["offers"]["highPrice"]) print(product["aggregateRating"]) ``` This pulls the JSON-LD block that [web-scraping.dev's product page](https://web-scraping.dev/product/1) embeds for search engines, which already contains the name, price range, and rating as clean typed fields instead of text scattered across the page. Check these articles for more about scraping hidden data. [How to Scrape Hidden Web DataThe visible HTML doesn't always represent the whole dataset available on the page. In this article, we'll be taking a look at scraping of hidden web data. What is it and how can we scrape it using Python?](https://scrapfly.io/blog/posts/how-to-scrape-hidden-web-data) [How to Scrape Hidden APIsIn this tutorial we'll be taking a look at scraping hidden APIs which are becoming more and more common in modern dynamic websites - what's the best way to scrape them?](https://scrapfly.io/blog/posts/how-to-scrape-hidden-apis) ## 5. Use Headless Browsers Only When JavaScript Rendering Demands It A headless browser is the right tool for JavaScript-heavy pages and the wrong default for everything else. Browser automation uses substantially more compute and memory than a plain HTTP request, and running it as a blanket default costs far more than most teams realize until the infrastructure bill arrives. The decision rule is simple to apply. Try a plain HTTP client first. If the response comes back as an empty shell and the data is not sitting in a hidden API or embedded JSON block like in the fourth rule, only then reach for Playwright, Puppeteer, or Selenium to render the page. The cost is not just speed: - Browser instances are memory-hungry - They run slower per request by a wide margin - They add another fingerprint surface that anti-bot systems can evaluate Running them "just in case" on pages that never needed JavaScript wastes money and increases detection surface for no benefit at all. The common failure is defaulting an entire pipeline to headless rendering on day one, only to discover months later that the scraper's compute bill costs more than the data it collects is worth. ## 6. Rotate Proxies, User Agents, and Fingerprints Responsibly Distribute the signals that identify a request, its IP, its User-Agent, its headers, across realistic pools so that no single identity accumulates suspicious volume on its own. The goal is spreading load, not disguising intent. Three layers work together here. 1. **IP rotation** through a proxy pool is the most visible one, where the tradeoff is datacenter IPs being cheaper and residential IPs looking more like real users. [How to Rotate Proxies in Web ScrapingIn this article we explore proxy rotation. How does it affect web scraping success and blocking rates and how can we smartly distribute our traffic through a pool of proxies for the best results.](https://scrapfly.io/blog/posts/how-to-rotate-proxies-in-web-scraping) 2. **User-Agent and header rotation** matters just as much, since a missing or stale User-Agent is an avoidable signal that costs nothing to fix. [How to Effectively Use User Agents for Web ScrapingIn this article, we’ll take a look at the User-Agent header, what it is and how to use it in web scraping. We'll also generate and rotate user agents to avoid web scraping blocking.](https://scrapfly.io/blog/posts/user-agent-header-in-web-scraping) 3. **Fingerprint consistency** ties both together, because a mismatch, like a modern User-Agent paired with an outdated TLS handshake, is exactly what detectors correlate to flag a request. [How Browser Fingerprinting Works and How to Defend Against ItLearn how browser fingerprinting works, from canvas to WebGPU, test your fingerprint with free tools, and apply developer-focused anti-detection strategies in your web scrapers.](https://scrapfly.io/blog/posts/how-browser-fingerprinting-works) Rotation exists to distribute load and avoid false-positive blocks on legitimate traffic, not to keep hitting a site that has already told you no. If a site blocks your identified scraper repeatedly, treat that response as the answer, which is exactly the framing practice 8 covers in more depth. Public-good crawlers go a step further and identify their purpose or a contact URL directly in the User-Agent, consistent with the crawler-identification example in RFC 9309 itself. One honeypot to avoid on the way: never follow a link hidden with `display: none` in CSS, a classic bot trap that a real browser user would never trigger. The common failure is rotating User-Agents while keeping one static datacenter IP, or the reverse, rotating IPs while sending the same stale header set on every request. Detectors correlate the layers, so partial rotation often signals automation more clearly than no rotation at all. Managed scraping APIs automate pool management and fingerprint consistency; Scrapfly handles [proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) automatically on every request. To understand more about proxies, check: [The Complete Guide To Using Proxies For Web ScrapingIntroduction to proxy usage in web scraping. What types of proxies are there? How to evaluate proxy providers and avoid common issues.](https://scrapfly.io/blog/posts/introduction-to-proxies-in-web-scraping) Scrapfly #### Scale your web scraping effortlessly Scrapfly handles proxies, browsers, and anti-bot bypass — so you can focus on data. [Try Free →](https://scrapfly.io/register)## 7. Handle Errors With Retries and Exponential Backoff Production scrapers assume failure by default. Classify every error as retryable or terminal, retry the retryable ones with exponentially growing waits plus jitter, and cap the attempt count so one bad URL never stalls the entire pipeline behind it. | Usually Retryable | Usually Terminal Until Corrected | |---|---| | Timeouts, connection resets | 404 Not Found | | 429 Too Many Requests | 401 or 403 auth failures | | 503 Service Unavailable | Parse or schema mismatch | RFC 9110 defines 429 as rate limiting and 503 as a temporary inability to serve the request, and both are worth retrying after a wait. Retrying the terminal column instead just burns requests and goodwill against a problem a retry can never fix. The backoff recipe: - Use capped exponential backoff with jitter - Honor a `Retry-After` header when the server sends one - Set a finite attempt cap for the workload AWS's architecture guidance on exponential backoff and jitter explains why jitter matters: it spreads retry spikes across many clients instead of letting them all retry in lockstep. Treat that as the reasoning to apply, not as specific numbers to copy into your own workload. Log everything as you go: URL, status code, latency, and retry count. That log is exactly what makes practice 11's monitoring possible later. python```python import random import time import httpx def fetch_with_retry(url, max_attempts=5): retryable = {429, 500, 502, 503, 504} for attempt in range(1, max_attempts + 1): response = httpx.get(url, timeout=10) if response.status_code < 400: return response if response.status_code not in retryable or attempt == max_attempts: response.raise_for_status() retry_after = response.headers.get("Retry-After") wait = float(retry_after) if retry_after else min(2 ** attempt, 30) + random.uniform(0, 1) time.sleep(wait) response = fetch_with_retry("https://web-scraping.dev/product/1") print(response.status_code) ``` The common failure is retrying a 429 immediately with no backoff at all, which increases load on an already-limiting server and can extend the block instead of clearing it. Managed APIs bake this in. Scrapfly's [scrape API](https://scrapfly.io/docs/scrape-api/getting-started) handles retries and failure classification automatically on every request. [What is HTTP Error 429 Too Many Request and How to Fix itHTTP 429 is an infamous response code that indicates request throttling or distribution is needed. Let's take a look at how to handle it.](https://scrapfly.io/blog/posts/what-is-http-error-429-too-many-requests) ## 8. Treat 403, 429, and CAPTCHA Responses as Signals, Not Challenges A block response is feedback about your scraper's behavior or the site's access policy, not an obstacle to force through. The professional response is to slow down and reconsider the approach, not to escalate against the same endpoint with the same configuration. Read each signal for what it actually says: - **429** means the server is applying rate limiting, so back off and honor `Retry-After` when it is present [What is HTTP Error 429 Too Many Request and How to Fix itHTTP 429 is an infamous response code that indicates request throttling or distribution is needed. Let's take a look at how to handle it.](https://scrapfly.io/blog/posts/what-is-http-error-429-too-many-requests) - **403** means the server understood the request but refuses to fulfill it, per RFC 9110, whether that comes from an access policy, IP reputation, or the request's own signals looking automated [How to Fix 403 Forbidden Errors When Web ScrapingLearn why web scrapers get 403 Forbidden errors and how to fix them with 7 Python solutions, from headers to TLS fingerprinting.](https://scrapfly.io/blog/posts/403-forbidden-web-scraping) - **CAPTCHA** means the site is actively challenging the traffic pattern it is seeing, which is a stronger statement than either status code alone [How to Know What Anti-Bot Service a Website is Using?In this article we'll take a look at two popular tools: WhatWaf and Wafw00f which can identify what WAF service is used.](https://scrapfly.io/blog/posts/how-to-know-what-anti-bot-website-uses) | Signal | Likely Cause | Respectful Response | |---|---|---| | 429 Too Many Requests | Rate limit reached | Honor `Retry-After`, reduce concurrency, slow down | | 403 Forbidden | Access policy, IP reputation, or request signals | Review identity practices from practice 6 before retrying | | CAPTCHA challenge | Traffic pattern flagged as automated | Slow down, escalate identity practices rather than brute force | | Block persists after fixes | The source does not want automated access | Stop and reassess whether this data source wants automated access at all | When the data genuinely matters and access is legitimate, the fixes are the identity practices from the sixth rule and the dedicated anti-bot guidance, not a technique taught here. The common failure is retrying a 403 over and over with the exact same configuration, which adds load without changing the reason the server is refusing the request in the first place. For production access to protected targets, Scrapfly's [anti-scraping protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) handles the challenge layer automatically. ## 9. Cache Responses and Deduplicate Requests Avoid fetching unchanged content twice when a cache or a validator already tells you it has not changed. Caching cuts cost, speeds up a pipeline, and reduces load on the target site at the same time, which makes it one of the few practices here with no real tradeoff. Store responses locally during development, since re-running a parser against live pages on every test is an avoidable waste. For revisit workloads, use conditional requests with `ETag` and `Last-Modified` (RFC 9110, sections 8.8 and 13.1), and set TTLs to match how often the data actually changes. Deduplication works alongside caching rather than replacing it. Normalize URLs before they enter the queue and honor a page's `rel="canonical"` tag so that desktop and mobile variants, or copies carrying different tracking parameters, do not get scraped twice as if they were separate pages. python```python import httpx with httpx.Client() as client: first = client.get("https://httpbin.dev/etag/scrapfly-demo") etag = first.headers["etag"] cached = client.get( "https://httpbin.dev/etag/scrapfly-demo", headers={"If-None-Match": etag}, ) print(cached.status_code) # 304, reuse the stored copy ``` The common failure is a crawler with no visited-URL set, repeatedly scraping the same catalog pages on every run and wasting proxy transfer for zero new data each time. [How to Use Cache In Web Scraping for Major Performance BoostIntroduction to web scraping caches. How caching can significantly reduce scraping costs and drastically improve performance.](https://scrapfly.io/blog/posts/how-to-use-cache-in-web-scraping) ## 10. Validate Data at Extraction and Store It With Provenance Parse and validate while you scrape, not after the fact. A large unvalidated batch can hide a large number of extraction problems behind a row count that looks perfectly normal. Validate at the point of extraction: schema-check each record as it is parsed confirming required fields are present, types are correct, and values fall inside a plausible range. A site redesign then surfaces as an immediate error spike, not a silent null flood found weeks later. Provenance is the other half. Store the source URL, fetch timestamp, and scraper version alongside every record, so a questioned data point can be traced and a targeted re-scrape replaces re-running the whole pipeline. In practice that means schema validators at the parse boundary with a quarantine bucket for failed records. The common failure is discovering much later that a redesign shifted a price field, with nothing having validated the parse or recorded which scraper version produced which rows. [What is Parsing? From Raw Data to InsightsLearn about the fundamentals of parsing data, across formats like JSON, XML, HTML, and PDFs. Learn how to use Python parsers and AI models for efficient data extraction.](https://scrapfly.io/blog/posts/what-is-parsing-turning-data-into-insights) [XPath vs CSS selectors: what's the difference?CSS selectors and XPath are both path languages for HTML parsing. Xpath is more powerful but CSS is more approachable - which is one is better?](https://scrapfly.io/blog/answers/xpath-vs-css-selectors) ## 11. Monitor Your Scrapers and Alert on Breakage Websites change without notice, so a production scraper needs the same observability as any other production service: success-rate metrics, data-volume baselines, and an alert when either one drifts from what is normal. Three signals are worth watching on every scraper you run in production: - **Request health**: success rate, the mix of status codes you are seeing, and latency - **Extraction health**: records per run and field fill rates measured against a baseline from healthy runs - **Freshness**: the timestamp of the last successful run per target, which catches a scraper that has silently stopped running altogether The cheap way to implement this is the logging already built in the seventh rule, paired with a scheduled check and one alert channel. A small healthchecks script that compares today's record count against last week's average catches most breakage far earlier than a manual spot-check ever will. Most checklists skip this practice, but selectors breaking and layout changes are a major failure class in any long-running scraper, and detection speed determines how much data you lose before someone notices. The common failure a good monitor catches: the scraper "succeeds," HTTP 200 on every request, while extracting zero records because a container div was quietly renamed. A records-per-run baseline flags that instantly, where a status-code check alone would miss it. ## FAQ Is web scraping legal?Public accessibility alone does not resolve whether a scraping project is lawful. Check applicable law, authentication boundaries, personal-data obligations, terms of service, and intended reuse. This is not legal advice, so consult a lawyer for commercial projects involving personal or gated data. Do scrapers still need to respect robots.txt in 2026?Yes, as a professional norm. robots.txt is not a technical enforcement mechanism, and the Duke University study published May 27, 2025 found that bot compliance drops as directives get stricter. Respecting the file makes your access expectations explicit and avoids intentionally adding load to paths an owner has marked off-limits. Can websites detect my scraper?Yes. Sites can evaluate request patterns, IP reputation, HTTP headers, TLS characteristics, browser fingerprints, or some combination of all of these signals at once. Exact detection methods vary by site and by which anti-bot vendor it runs, and the dedicated [5 Tools to Scrape Without Blocking and How it All WorksTutorial on how to avoid web scraper blocking. What is javascript and TLS (JA3) fingerprinting and what role request headers play in blocking.](https://scrapfly.io/blog/posts/how-to-scrape-without-getting-blocked-tutorial) guide covers those layers in depth. Which websites allow scraping practice?Purpose-built sandboxes exist for exactly this. [web-scraping.dev](https://web-scraping.dev) describes scenarios for pagination, authentication, GraphQL APIs, and other patterns you will meet in production. Practicing against a site designed for learning is safer than probing an unrelated live site, and its published terms still apply even though it was built for practice. ## Summary Best practices are failure prevention, not etiquette. This checklist pays for itself the first time a site changes under you without warning. Plan against the site's rules first, acquire data through the most stable channel available, assume failure at every step, and watch your pipeline like any other production service. For teams that want these reliability practices, rotation, retries, caching, anti-bot handling, managed for them, Scrapfly's [Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started) automates them behind a single request, with SDKs for Python, TypeScript, Go, and Rust. A disciplined DIY pipeline built on this checklist works just as well for teams that would rather own every layer themselves. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [Why Do Most Web Scrapers Fail Without Best Practices?](#why-do-most-web-scrapers-fail-without-best-practices) - [1. Check for an Official API or Bulk Export Before Scraping](#1-check-for-an-official-api-or-bulk-export-before-scraping) - [2. Respect robots.txt, Crawl-Delay, and the Site's Terms](#2-respect-robots-txt-crawl-delay-and-the-site-s-terms) - [3. Rate Limit Requests and Scrape During Off-Peak Hours](#3-rate-limit-requests-and-scrape-during-off-peak-hours) - [4. Prefer Hidden APIs and Embedded JSON Over HTML Parsing](#4-prefer-hidden-apis-and-embedded-json-over-html-parsing) - [5. Use Headless Browsers Only When JavaScript Rendering Demands It](#5-use-headless-browsers-only-when-javascript-rendering-demands-it) - [6. Rotate Proxies, User Agents, and Fingerprints Responsibly](#6-rotate-proxies-user-agents-and-fingerprints-responsibly) - [7. Handle Errors With Retries and Exponential Backoff](#7-handle-errors-with-retries-and-exponential-backoff) - [8. Treat 403, 429, and CAPTCHA Responses as Signals, Not Challenges](#8-treat-403-429-and-captcha-responses-as-signals-not-challenges) - [9. Cache Responses and Deduplicate Requests](#9-cache-responses-and-deduplicate-requests) - [10. Validate Data at Extraction and Store It With Provenance](#10-validate-data-at-extraction-and-store-it-with-provenance) - [11. Monitor Your Scrapers and Alert on Breakage](#11-monitor-your-scrapers-and-alert-on-breakage) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ http python ### Web Scraping with Python Introduction tutorial to web scraping with Python. How to collect and parse public data. Challenges, best practices and ... ](https://scrapfly.io/blog/posts/web-scraping-with-python) [ python ### Everything to Know to Start Web Scraping in Python Today Complete introduction to web scraping using Python: http, parsing, AI, scaling and deployment. ](https://scrapfly.io/blog/posts/everything-to-know-about-web-scraping-python) [ http nodejs ### Web Scraping With NodeJS and Javascript In this article we'll take a look at scraping using Javascript through NodeJS. We'll cover common web scraping libraries... ](https://scrapfly.io/blog/posts/web-scraping-with-nodejs) Scale your web scraping effortlessly, **1,000 free credits** [Start Free](https://scrapfly.io/register)