[Blog](https://scrapfly.io/blog) / [data-parsing](https://scrapfly.io/blog/tag/data-parsing) / [How to Test a Web Scraper Before Production](https://scrapfly.io/blog/posts/how-to-test-web-scraper-before-production) # How to Test a Web Scraper Before Production by [Mayada Shaaban](https://scrapfly.io/blog/author/mayada-shaaban-90143e67) Oct 16, 2026 24 min read [\#data-parsing](https://scrapfly.io/blog/tag/data-parsing) [\#python](https://scrapfly.io/blog/tag/python) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-test-web-scraper-before-production "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-test-web-scraper-before-production&text=How%20to%20Test%20a%20Web%20Scraper%20Before%20Production "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-test-web-scraper-before-production "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-test-web-scraper-before-production) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-test-web-scraper-before-production) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-test-web-scraper-before-production) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-test-web-scraper-before-production) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-test-web-scraper-before-production) A scraper that returns five product cards from one page has only passed a demo. Change the layout, remove a field, repeat the final cursor, or return a login page with HTTP 200, and the same green run can save empty or duplicate records. The fix is to pull the parser away from the network and build fixtures for the variants that break scrapers. Add one bounded live check on top, and bad output stops a deployment before it reaches storage. [Web Scraping with PythonIntroduction tutorial to web scraping with Python. How to collect and parse public data. Challenges, best practices and an example project.](https://scrapfly.io/blog/posts/web-scraping-with-python) ## Key Takeaways - **A green run covers one input, once.** Other layouts stay untested. - **Separate fetching from parsing.** Parser tests read local fixtures only. - **Build fixtures from real page shapes**, then redact credentials before storing them. - **Assert the output contract** after HTTP 200: required fields, types, and unique IDs. - **Run one small live contract test** for drift, marked so downtime won't fail every build. - **Block deployment on failed fixtures or repeated cursors**, and hold it on a dead target. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## What Does One Successful Web Scraper Run Fail to Prove? A green run means one input made it through the current code path once. Other valid layouts, state transitions, and documents that should fail were never exercised. Five failure modes separate a working demo from a scraper that follows [production web scraping best practices](https://scrapfly.io/blog/posts/web-scraping-best-practices): - **Alternate layout returns different fields.** The parser works on the sampled product card but misses a sale, out-of-stock, or missing-image variant. - **Pagination never terminates.** A repeated cursor or last-page link causes duplicate IDs or an infinite loop. This one burns the most requests before anyone notices, which is why `crawl_listing` further down tracks visited URLs and caps the page count. - **Session state changes the document.** Login, consent, locale, or currency state returns a valid page that is wrong for the extraction job. - **HTTP 200 carries the wrong body.** A challenge, empty shell, or interstitial passes the status check and fails the content contract. - **The target changes after fixtures were saved.** Local tests stay green while the live selectors stop matching. A January 2026 [r/webscraping thread](https://www.reddit.com/r/webscraping/comments/1qjs0sz/how_do_you_verify_what_you_scrape/)asked how to verify that 10,000 scraped news items still match their source. The answer has two parts. Fixtures help with half of that. They tell you the parser reads the pages you saved, but whether a crawl found every item on a live site is a counting problem, and no fixture can enumerate a moving target. Each failure above needs the cheapest test that can observe it, and three test layers split the work. ## How Should Parser Unit Tests, HTTP Integration Tests, and Live Contract Tests Split the Work? Parser tests run a pure function against saved HTML and never open a socket. HTTP tests go one step out, so the crawler runs for real against responses you wrote. Only the live test touches the real site, to check that it still looks like your fixtures. | Test layer | Input | Network | Proves | Does not prove | |---|---|---|---|---| | Parser unit test | Local HTML fixture | No | Selectors, normalization, required output | Current live HTML | | HTTP integration test | Mock response or local server | Controlled | Redirects, statuses, retries, response routing | Target availability or current layout | | Live contract test | Small known URL set | Yes | Current upstream shape and access path | Full-site completeness or future stability | | Production check | Real job summary | Yes | Post-deployment drift in records and fields | Root cause by itself | Don't make a parser test fetch anything. `requests` has its own test suite, and a saved page reproduces the failure in milliseconds with no network in the loop. You still need one live contract, because a fixture freezes the page at capture time. Only a call to the real site notices that the target moved on. A 2016 [r/rails thread](https://www.reddit.com/r/rails/comments/5ewp6j/how_to_properly_test_when_web_scraping/)reached a similar split from the Ruby side. Its top answer says to test the parsing logic, mock everything else, and record responses with VCR so an outage can't fail the suite. Each dashed line back to the saved response means a failing layer sends you back to capture the page that broke it. Every regression test starts from one of those pages. [Controlled web scraping test sites](https://scrapfly.io/blog/posts/best-websites-to-practice-web-scraping) are a good place to capture those documents before you point the suite at a real target. ## How Do You Build HTML Fixtures for Web Scraper Regression Tests? A fixture corpus is a handful of real responses, each trimmed to the markup that reproduces one bug, with the expected parsed record stored beside it. For this example we captured responses from `web-scraping.dev/products`, then cut away most of the irrelevant markup while keeping the `.row.product`, `h3 a`, and `.price`structure intact. The file names say what each one tests: - `product_normal.html` - `product_missing_optional_field.html` - `listing_empty.html` - `listing_next_page.html` - `response_login_interstitial.html` - `response_consent_interstitial.html` - `response_empty_shell.html` - `response_rate_limited.html` The expected Python record sits next to each fixture, so a failed test is easy to read without reopening the HTML. Don't cut a fixture down to three perfect elements, though. At that point you're mostly testing Beautiful Soup. Leave enough of the page that a bad selector fails the same way it would on the real site. | Fixture | Behavior represented | Expected result | Failure class | |---|---|---|---| | `product_normal.html` | Two ordinary product cards | 2 records, no `stock_note` | None | | `product_missing_optional_field.html` | One card carries a stock badge, one doesn't | 2 records, one `stock_note` set | None | | `listing_empty.html` | Filtered search with zero matches | `[]` | None | | `listing_next_page.html` | Page 2 with a `rel="next"` link | 1 record, `next_url` set | None | | `response_login_interstitial.html` | Session lost, redirected to login | Raises `ParserError` | Wrong document | | `response_consent_interstitial.html` | Consent banner instead of listing | Raises `ParserError` | Wrong document | | `response_empty_shell.html` | No `.products` container at all | Raises `ParserError` | Wrong document | | `response_rate_limited.html` | 429 response body | Not parsed, handled by status check | Rate limited | web-scraping.dev has no stock badge, so `product_missing_optional_field.html` is the one fixture edited by hand. It holds two trimmed cards from the live page, and one extra `.stock-note` element on the first card to exercise the optional branch: html ```html

Teal Energy Potion

Low Stock
4.99

Red Energy Potion

4.99
``` The other fixtures the tests load are shorter. `product_normal.html` holds the first two cards from `web-scraping.dev/products` in the same trimmed shape, `listing_empty.html` is just `
`, and `response_login_interstitial.html` only needs a page containing `
`. Redact credentials, authorization headers, cookies, personal data, and tracking tokens before committing any captured page or recorded HTTP exchange to the repository. When a bug gets fixed, the response that caused it goes into the corpus first. Write the expected record, then change the parser until that fixture passes. Old fixtures stay as they were captured, even after the target changes. They record what the site used to send, and a new layout gets a new file. ## Project Setup The parser tests need [pytest](https://docs.pytest.org/en/stable/) and `beautifulsoup4`, the live contract test uses `requests`, and the schema section uses `pydantic`. Install all four before writing any tests: bash ```bash pip install pytest beautifulsoup4 requests pydantic ``` The tests live in `tests/` but import `scraper.py` and `crawl.py` from the project root. Put a `pytest.ini` in the root that adds it to the import path, or `pytest` stops at collection with `ModuleNotFoundError: No module named 'scraper'`: ini ```ini [pytest] pythonpath = . ``` The parser comes first, since every later test calls it. ## How Do You Unit Test Web Scraper Output With pytest? Pass local HTML into a pure parser function and use [pytest.mark.parametrize](https://docs.pytest.org/en/stable/how-to/parametrize.html)to run the same output contract against several page variants. The project has three parts: `scraper.py`, `tests/fixtures/`, and `tests/test_parser.py`. `parse_products(html)` stays independent of the URL, retry policy, or proxy configuration, so it never needs a network connection to run: python ```python # scraper.py import re from bs4 import BeautifulSoup class ParserError(Exception): """Raised when a document is not a valid product listing.""" def parse_products(html: str) -> list[dict]: soup = BeautifulSoup(html, "html.parser") container = soup.select_one(".products") cards = soup.select(".row.product") if not cards: if soup.select_one("#login-form") or soup.select_one(".consent-banner"): raise ParserError("document is a login or consent page, not a listing") if container is None: raise ParserError("no product listing container; treat as an empty shell") return [] products, seen_ids = [], set() for card in cards: link = card.select_one("h3 a") price_el = card.select_one(".price") if link is None or price_el is None: raise ParserError("product card is missing a title or price element") product_id = int(re.search(r"/product/(\d+)", link["href"]).group(1)) if product_id in seen_ids: raise ParserError(f"duplicate product id {product_id} in one document") seen_ids.add(product_id) note_el = card.select_one(".stock-note") products.append({ "id": product_id, "title": link.get_text(strip=True), "url": link["href"], "price": float(price_el.get_text(strip=True)), "stock_note": note_el.get_text(strip=True) if note_el else None, }) return products ``` The listing container check matters as much as the selectors. Without it, a blank page with no `.row.product` elements and no `.products` wrapper returns `[]`, and `[]` looks the same as a search with zero results. This test only checks that the fields exist, so a price of `-1` would pass. The Pydantic model further down rejects it, and [validate scraped field types and formats](https://scrapfly.io/blog/posts/how-to-ensure-web-scrapped-data-quality)goes further into coercion and normalization. Parametrize the test across normal, missing-optional-field, and empty-listing fixtures, plus a login page that has to raise `ParserError`: python ```python # tests/test_parser.py from pathlib import Path import pytest from scraper import ParserError, parse_products FIXTURES = Path(__file__).parent / "fixtures" def load(name): return (FIXTURES / name).read_text() @pytest.mark.parametrize("fixture_name,expected", [ ("product_normal.html", [ {"id": 1, "title": "Box of Chocolate Candy", "price": 24.99, "stock_note": None}, {"id": 2, "title": "Dark Red Energy Potion", "price": 4.99, "stock_note": None}, ]), ("product_missing_optional_field.html", [ {"id": 3, "title": "Teal Energy Potion", "price": 4.99, "stock_note": "Low Stock"}, {"id": 4, "title": "Red Energy Potion", "price": 4.99, "stock_note": None}, ]), ("listing_empty.html", []), ]) def test_parse_products_matches_expected_records(fixture_name, expected): products = parse_products(load(fixture_name)) trimmed = [{"id": p["id"], "title": p["title"], "price": p["price"], "stock_note": p["stock_note"]} for p in products] assert trimmed == expected def test_parse_products_rejects_login_interstitial_as_wrong_document(): with pytest.raises(ParserError): parse_products(load("response_login_interstitial.html")) ``` Running `pytest -v` prints four passing lines: text ```text tests/test_parser.py::test_parse_products_matches_expected_records[product_normal.html-expected0] PASSED tests/test_parser.py::test_parse_products_matches_expected_records[product_missing_optional_field.html-expected1] PASSED tests/test_parser.py::test_parse_products_matches_expected_records[listing_empty.html-expected2] PASSED tests/test_parser.py::test_parse_products_rejects_login_interstitial_as_wrong_document PASSED ``` The bracketed suffix is pytest's parametrize ID, so a failure names the fixture that broke it. [How to Parse Web Data with Python and BeautifulsoupBeautifulsoup is one the most popular libraries in web scraping. In this tutorial, we'll take a hand-on overview of how to use it, what is it good for and explore a real -life web scraping example.](https://scrapfly.io/blog/posts/web-scraping-with-python-beautifulsoup) Pagination, sessions, and HTTP failures live outside `parse_products`. Testing them means controlling what `fetch` returns. ## How Do You Test Pagination, Session Cookies, and HTTP Failures? Patch one function, `fetch`, and the whole crawler runs against responses you wrote, including its navigation, session state, and stop conditions. [pytest's monkeypatch](https://docs.pytest.org/en/stable/how-to/monkeypatch.html)fixture or the standard library's [unittest.mock](https://docs.python.org/3/library/unittest.mock.html) both swap out the fetch dependency for the test's duration. Patch `crawl.fetch`, the name the calling module looks up at call time. `monkeypatch` undoes the change when the test finishes, and `monkeypatch.setattr` raises `AttributeError` if `crawl.py` has no `fetch` to replace, so a typo in the patch target fails loudly instead of patching nothing. `crawl_listing` calls `fetch(url, session)` by name at the module level, so `monkeypatch.setattr(crawl, "fetch", ...)` redirects every call inside it: python ```python # crawl.py from dataclasses import dataclass from urllib.parse import urljoin from bs4 import BeautifulSoup from scraper import ParserError, parse_products @dataclass class Response: status_code: int text: str next_url: str | None = None def fetch(url, session): response = session.get(url, timeout=10) paging = BeautifulSoup(response.text, "html.parser").select(".paging a") next_href = paging[-1].get("href") if paging else None # the ">" link has no href on the last page next_url = urljoin(response.url, next_href) if next_href else None return Response(response.status_code, response.text, next_url) class BlockedError(Exception): pass class RateLimitedError(Exception): pass def crawl_listing(start_url, session, max_pages=20): products, seen_ids, visited = [], set(), set() url, pages_fetched = start_url, 0 while url and pages_fetched < max_pages: if url in visited: break # repeated cursor: stop instead of looping forever visited.add(url) pages_fetched += 1 response = fetch(url, session) if response.status_code == 429: raise RateLimitedError(f"rate limited fetching {url}") if response.status_code == 403: raise BlockedError(f"blocked fetching {url}") try: page_products = parse_products(response.text) except ParserError as exc: raise ParserError(f"{url} did not return the expected product listing: {exc}") from exc for product in page_products: if product["id"] not in seen_ids: seen_ids.add(product["id"]) products.append(product) url = response.next_url return products ``` `max_pages` is the backstop for a site that mints a fresh cursor on every page, where `visited` never sees a repeat. On web-scraping.dev the last page still renders the `>` link, but without an `href`, so `fetch` checks the `href` itself. ### How Should Pagination Tests Catch Repeated Cursors and Duplicate IDs? Test a normal two-page sequence, a final page with no next link, a repeated next URL, and the same product ID appearing on two pages. Assert that the crawl ends and IDs stay unique. Page and record counts break the first time the catalogue changes: python ```python from pathlib import Path from unittest.mock import Mock import pytest import crawl from crawl import BlockedError, crawl_listing from scraper import ParserError FIXTURES = Path(__file__).parent / "fixtures" def load(name): return (FIXTURES / name).read_text() def make_response(status_code=200, text="", next_url=None): return crawl.Response(status_code=status_code, text=text, next_url=next_url) def test_crawl_listing_stops_on_a_repeated_next_url(monkeypatch): looping_url = "https://web-scraping.dev/products?page=1" monkeypatch.setattr(crawl, "fetch", Mock(return_value=make_response( text=load("product_normal.html"), next_url=looping_url, ))) crawl_listing(looping_url, session={}) assert crawl.fetch.call_count == 1 ``` A two-page sequence that repeats both product IDs covers the final page and deduplication in one test: python ```python def test_crawl_listing_follows_two_pages_and_drops_a_repeated_id(monkeypatch): page_2 = "https://web-scraping.dev/products?page=2" monkeypatch.setattr(crawl, "fetch", Mock(side_effect=[ make_response(text=load("product_normal.html"), next_url=page_2), make_response(text=load("product_normal.html")), # same IDs, no next link ])) products = crawl_listing("https://web-scraping.dev/products", session={}) assert crawl.fetch.call_count == 2 ids = [p["id"] for p in products] assert len(ids) == len(set(ids)) ``` The second page has no next link, so the crawl must stop there, and the test fails as soon as deduplication breaks. Neither test covers every scheme. A real catalogue might use opaque cursors, numbered pages, or changing query parameters. When a new pattern breaks the crawl, save it as another case. ### How Should Session-Cookie Tests Cover Login, Consent, and Locale? Return the login document through the same fetch boundary and check the document type before trusting a 200. Consent and wrong-locale pages each get one test in the same shape, with their own fixture: python ```python def test_crawl_listing_fails_on_login_instead_of_returning_empty(monkeypatch): monkeypatch.setattr(crawl, "fetch", Mock(return_value=make_response( text=load("response_login_interstitial.html"), ))) with pytest.raises(ParserError): crawl_listing("https://web-scraping.dev/account/products", session={}) ``` A successful status code with a login or consent page in the body has to fail as the wrong document. An empty product list there would look like a search with zero results. Keep authentication secrets out of both fixtures and test logs. ### How Should HTTP 200, HTTP 403, and HTTP 429 Fixtures Fail? Give each status code its own tested outcome. One shared fallback hides which one fired: - **200 target document:** parse and return records. - **200 interstitial or empty shell:** reject before storage, since the listing never arrived. - **403:** treat as terminal for the current request configuration and stop. Looping blindly on a 403 burns requests. - **429:** surface rate limiting to the caller so the application's own retry policy decides what happens next. python ```python def test_crawl_listing_raises_blocked_on_403_without_retrying(monkeypatch): monkeypatch.setattr(crawl, "fetch", Mock(return_value=make_response(status_code=403))) with pytest.raises(BlockedError): crawl_listing("https://web-scraping.dev/products", session={}) assert crawl.fetch.call_count == 1 ``` When we ran them with pytest 9.1, the offline tests in this article finished in well under a tenth of a second. Bounded retries and error handling belong in this layer, with one branch for each status code you expect. Scrapfly #### Extract structured data automatically? Scrapfly's Extraction API uses AI to turn any webpage into structured data — no selectors needed. [Try Free →](https://scrapfly.io/register)## Which Output Schema and Pydantic Assertions Catch Silent Web Scraper Failures? Assert the fields that make a record usable, and reject or quarantine any record that fails. Getting a 200 is only the start. The response might be the right page, and your selector can still latch onto the wrong element and quietly produce a zero price, an empty title, or some unrelated text. A record needs a stable ID or canonical URL, a title, and the job-specific field the consumer needs, price in this case. Types matter too. A price has to parse as a number, optional fields need a recognized shape, and no field should carry raw HTML where plain text belongs. IDs must stay unique across the page sequence. Zero records should come from the `listing_empty.html` path and nowhere else. A compact [Pydantic](https://pydantic.dev/docs/validation/latest/get-started/) model puts those rules in one place: python ```python from pydantic import BaseModel, Field, HttpUrl, PositiveFloat class Product(BaseModel): id: int title: str = Field(min_length=1) url: HttpUrl price: PositiveFloat stock_note: str | None = None ``` `Product(**record)` raises a `ValidationError` the moment a title comes back empty or a price comes back negative or zero, both signs of a parser reading the wrong element. A full validation library goes further, with coercion and normalization the dedicated data-quality guide already covers. That r/webscraping question about 10,000 news items gets half its answer here. A schema check rejects a malformed record on sight. Counting what the crawl missed is a job for production monitoring. Print the rejected fixture name and the failure reason in test output, beyond a plain pass or fail count, so whoever picks up the failure can rerun that one branch. ## When Should Live Contract Tests Call web-scraping.dev or the Target Website? Live calls stay out of the default run. Mark the test with `@pytest.mark.live` and register the marker in `pytest.ini`, so developers and CI skip it unless they ask: ini ```ini [pytest] markers = live: calls the real target over the network; excluded from the default run pythonpath = . addopts = -m "not live" ``` Plain `pytest` now deselects anything marked `live`, and `pytest -m live` runs only those tests. The public example uses `web-scraping.dev/products` and the [requests](https://requests.readthedocs.io/en/latest/) library. It asserts on structure, because catalogue content changes and a specific title would break the test: python ```python # tests/test_live_contract.py import pytest import requests from scraper import parse_products LIVE_URL = "https://web-scraping.dev/products" @pytest.mark.live def test_live_contract_products_page_matches_the_saved_shape(): response = requests.get(LIVE_URL, timeout=10) assert response.url.startswith("https://web-scraping.dev") assert "text/html" in response.headers.get("content-type", "") assert response.status_code == 200 products = parse_products(response.text) assert len(products) > 0 assert all(p["title"] and p["url"] and p["price"] > 0 for p in products) ``` Against the live page, this test passed in about a second when we ran it. For a real target, a practical live set stays small, with a normal detail page, an optional-field variant, an empty state, and one navigation boundary if the target allows it. How often you run it depends on the site. Run it on every deployment, and on a schedule that matches how often the target changes and what a missed change would cost. A marketplace that redesigns every quarter needs it far more often than a static archive does. When this test fails, save what you'll need to explain it: the requested URL, the final URL after redirects, the status code, and which selector stopped matching. A timeout points at the target, which is down or blocking you. If a selector misses, the parser is out of date, and the fix starts with a new fixture. The same [web scraping test sites](https://scrapfly.io/blog/posts/best-websites-to-practice-web-scraping)that work for building fixtures also work for this bounded check. When the live check fails, the first job is getting a copy of the response that broke it. ## How Can Scrapfly Debug and Replay a Failing Web Scrape? Turn on Scrapfly debug for the failing request to preserve the response, then replay extraction against the stored content and turn the saved page into a new regression fixture before touching the parser. With `debug=true`, the [Scrapfly debug](https://scrapfly.io/docs/scrape-api/debug) parameter stores the scraped result for the account's log-retention window and returns `response_url` plus `content_replay_url`. Add `render_js=true` and debug also attempts a screenshot, returned as `screenshot_url`. Reproduce the failure once with debug enabled. Debug adds 2 API credits per request in the live environment and is free in the test environment, so keep it off for full runs. Inspect the returned document, redact it, and drop it into the fixture corpus. Then write the assertion that should pass, before changing a line of the parser. For a response captured somewhere other than Scrapfly, the [Classify API](https://scrapfly.io/docs/scrape-api/classify) takes a `POST` to `/classify` with the final URL, status code, headers, and body. It returns a `blocked` boolean that separates a detected block page from an ordinary parser mismatch. Treat `blocked` as a hint, and confirm the cause before acting on it. For JavaScript-rendered paths, `render_js` combined with a target-specific `wait_for_selector`, or an `xhr:` prefixed selector for a background request, confirms the data loaded before the assertion runs. A fixed `sleep` passes on a fast day and flakes on a slow one. The code here stays in Python. The debug and replay parameters work the same through the [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Go](https://scrapfly.io/docs/sdk/golang), and [Rust](https://scrapfly.io/docs/sdk/rust) SDKs. ## Which Test Results Should Block Web Scraper Deployment? Most red results block the release. The exception is a dead target, which holds it until you know whether the code or the site broke: | Result | Deploy? | Required action | |---|---|---| | Parser fixture fails | No | Fix the parser, or update an intentionally changed contract with review | | Pagination repeats a cursor or ID | No | Fix termination or deduplication behavior | | HTTP 200 fixture emits records from an interstitial | No | Add document acceptance before parsing | | Live target unavailable | Hold and classify | Separate target or access failure from a code regression | | Live contract shows changed selectors | No | Save the new response, add a fixture, then update the parser | | Non-load-bearing optional field absent and contract allows it | Yes | Preserve the explicit optional behavior | Every failure in that table needs a reproducible artifact behind it: the fixture name, the URL class, the typed exception, and a redacted copy of the response. A red test with no attached artifact repeats the same debugging cycle next week. A green suite is a record of the failures you've already met. Whether the target stays online, or whether an open-ended crawl found every page, is outside anything these tests look at. After deployment, keep the live contract and output checks running on the schedule you picked for the target. Retry, caching, and monitoring design is its own topic, covered in [production web scraping best practices](https://scrapfly.io/blog/posts/web-scraping-best-practices). ## Power-up with Scrapfly ScrapFly's [Web Scraping API](https://scrapfly.io/products/web-scraping-api) is a single HTTP endpoint for collecting web data at scale, with a **98% success rate** on the hardest public pages and proxies across **190+ countries**. - [Web Unblocker](https://scrapfly.io/products/unblocker) (previously ASP) - handles 25+ anti-bot systems, including Cloudflare, DataDome, PerimeterX and Akamai. - [Smart proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) - residential and datacenter pools with country-level geo-targeting. - [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - render SPAs and dynamic pages through real cloud browsers. - [Browser automation scenarios](https://scrapfly.io/docs/scrape-api/javascript-scenario) - scroll, click, fill forms, and wait for elements without managing a browser fleet. - [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - return pages as HTML, JSON, clean text, or LLM ready Markdown. - [Session management](https://scrapfly.io/docs/scrape-api/session) - keep cookies, headers, and IPs consistent across multi step flows. - [Smart caching](https://scrapfly.io/docs/scrape-api/getting-started#api_param_cache) - cache successful responses to cut cost on repeat scraping jobs. - [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Go](https://scrapfly.io/docs/sdk/golang), [Rust](https://scrapfly.io/docs/sdk/rust), and [no-code integrations](https://scrapfly.io/docs/integration/getting-started) including [Make](https://scrapfly.io/integration/make), [n8n](https://scrapfly.io/integration/n8n), [Zapier](https://scrapfly.io/integration/zapier), [LangChain](https://scrapfly.io/integration/langchain), and [LlamaIndex](https://scrapfly.io/integration/llamaindex). Debug, replay, and rendering all run through that same endpoint, so capturing the failing page for a fixture uses the tool that already fetches it. ### Web Scraping API Scrape any website with our powerful API. Anti-bot bypass, JavaScript rendering, and rotating proxies built-in. [Try Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started) ## FAQ Should Web Scraper Unit Tests Call Live Websites?No. Parser unit tests should run entirely from local HTML fixtures, and a separately marked live contract test watches for upstream drift. A live-only suite mixes up target downtime or a block with a code regression. How Often Should a Live Scraper Contract Test Run?Run it on deployment and on a schedule matched to how often the target changes and what missed drift costs. A busy marketplace and a stable public archive carry different risk, so skip any universal daily or hourly rule. Should You Regenerate HTML Fixtures Automatically?No. Capture can be automated, but replacing an expected fixture needs review, since the changed page might be the regression the test was written to catch. Save a new variant as an additional fixture. Can pytest Test Playwright Scrapers?Yes. Test extraction logic against local rendered-HTML fixtures, then add a smaller [Playwright](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python) test for navigation and browser state. Assert on the locator that holds the data, because a finished `page.goto()`call isn't a success signal. ## Summary Real responses become fixtures, and the parser runs against them in milliseconds. A patched `fetch` covers pagination, sessions, and status codes. On top sits one live check for the day the real page changes. Together they keep a login page served with 200, a looping cursor, or an empty title from passing as a good deploy. A redesign that lands after release shows up in the live check first. If capturing and replaying the failing response, or reproducing a JavaScript-rendered page, is the hard part, Scrapfly debug, Classify, and JavaScript rendering feed that same fixture workflow. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents - [Key Takeaways](#key-takeaways) - [What Does One Successful Web Scraper Run Fail to Prove?](#what-does-one-successful-web-scraper-run-fail-to-prove) - [How Should Parser Unit Tests, HTTP Integration Tests, and Live Contract Tests Split the Work?](#how-should-parser-unit-tests-http-integration-tests-and-live-contract-tests-split-the-work) - [How Do You Build HTML Fixtures for Web Scraper Regression Tests?](#how-do-you-build-html-fixtures-for-web-scraper-regression-tests) - [Project Setup](#project-setup) - [How Do You Unit Test Web Scraper Output With pytest?](#how-do-you-unit-test-web-scraper-output-with-pytest) - [How Do You Test Pagination, Session Cookies, and HTTP Failures?](#how-do-you-test-pagination-session-cookies-and-http-failures) - [How Should Pagination Tests Catch Repeated Cursors and Duplicate IDs?](#how-should-pagination-tests-catch-repeated-cursors-and-duplicate-ids) - [How Should Session-Cookie Tests Cover Login, Consent, and Locale?](#how-should-session-cookie-tests-cover-login-consent-and-locale) - [How Should HTTP 200, HTTP 403, and HTTP 429 Fixtures Fail?](#how-should-http-200-http-403-and-http-429-fixtures-fail) - [Which Output Schema and Pydantic Assertions Catch Silent Web Scraper Failures?](#which-output-schema-and-pydantic-assertions-catch-silent-web-scraper-failures) - [When Should Live Contract Tests Call web-scraping.dev or the Target Website?](#when-should-live-contract-tests-call-web-scraping-dev-or-the-target-website) - [How Can Scrapfly Debug and Replay a Failing Web Scrape?](#how-can-scrapfly-debug-and-replay-a-failing-web-scrape) - [Which Test Results Should Block Web Scraper Deployment?](#which-test-results-should-block-web-scraper-deployment) - [Power-up with Scrapfly](#power-up-with-scrapfly) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ python ai ### What is Parsing? From Raw Data to Insights Learn about the fundamentals of parsing data, across formats like JSON, XML, HTML, and PDFs. Learn how to use Python par... ](https://scrapfly.io/blog/posts/what-is-parsing-turning-data-into-insights) [ http python ### Web Scraping with Python Introduction tutorial to web scraping with Python. How to collect and parse public data. Challenges, best practices and ... ](https://scrapfly.io/blog/posts/web-scraping-with-python) [ python ### Everything to Know to Start Web Scraping in Python Today Complete introduction to web scraping using Python: http, parsing, AI, scaling and deployment. ](https://scrapfly.io/blog/posts/everything-to-know-about-web-scraping-python) Extract structured data with AI, **1,000 free credits** [Start Free](https://scrapfly.io/register)