[Blog](https://scrapfly.io/blog)   /  [HTTP Referer Header: Complete Guide for Web Scraping](https://scrapfly.io/blog/posts/http-referer-header-complete-guide-for-web-scraping)   # HTTP Referer Header: Complete Guide for Web Scraping

 by [Hisham Medhat](https://scrapfly.io/blog/author/hisham) Jul 07, 2026 15 min read [  ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhttp-referer-header-complete-guide-for-web-scraping "Share on LinkedIn")    

 

 

         

   **Web Scraping API**Scrape any website with anti-bot bypass, proxy rotation, and JS rendering.

 

 [ Learn More  ](https://scrapfly.io/products/web-scraping-api) [  Docs ](https://scrapfly.io/docs/scrape-api/getting-started) 

 

 

Your scraper worked on the homepage. Then the product API returned a 422 or 403. The missing piece often is not JavaScript, cookies, or a proxy. The missing piece is a `Referer` value that matches how a real browser reached that endpoint.

In this guide, we'll cover what the HTTP Referer header does and how websites use it to detect scrapers. We'll also show why a fake `google.com` value usually makes things worse.

We'll show how to set Referer correctly in [requests](https://scrapfly.io/blog/posts/python-requests-headers-guide), [httpx](https://scrapfly.io/blog/posts/web-scraping-with-python-httpx), [Scrapy](https://scrapfly.io/blog/posts/web-scraping-with-scrapy), [Playwright](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python), [Puppeteer](https://scrapfly.io/blog/posts/web-scraping-with-puppeteer-and-nodejs), and Scrapfly. Let's get started!

[How Headers Are Used to Block Web Scrapers and How to Fix ItIntroduction to web scraping headers - what do they mean, how to configure them in web scrapers and how to avoid being blocked.](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-headers)



## Key Takeaways

Context matters more than any generic Referer value.

- The `Referer` header tells the server which page the client came from before the current request.
- A missing Referer on deep pages is suspicious, but a contextually wrong Referer is often worse.
- Modern browsers default to `strict-origin-when-cross-origin`, so cross-origin requests usually send only the origin, not the full path.
- Some APIs are referer-gated, which means the correct Referer is required to get a `200 OK` at all.
- Browser automation usually keeps a natural Referer chain on its own, but direct API calls still need manual overrides.
- Scrapfly can tune Referer automatically with `asp=True`, but deterministic referer-gated APIs still need the exact header you saw in the browser.

**Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime.







## What Is the HTTP Referer Header?

The HTTP `Referer` request header tells a server which URL the client was on before making the current request. If a browser moves from a listing page to a product page, the second request can include the listing URL as the Referer value.

That makes Referer useful for analytics, access control, and bot detection. A server can check whether the current request fits a believable navigation path, or whether the client appears to jump straight into an internal endpoint without the page that normally triggers it.

http```http
Referer: https://example.com/products/
```



The one-line example above is enough to understand the shape of the header. The key question is not how to spell the field or where to place it. The key question is whether the value matches a real browsing step.

### Why the header is misspelled

The field name is officially `Referer`, even though the English word is "referrer". MDN makes the same point in the [Referrer-Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy) docs. The request header kept the original misspelling, while the policy header did not.

That is why developers search for both "referer header" and "referrer header". In code, docs, and browser devtools, you should still use the official HTTP spelling: `Referer`.

### When browsers send Referer and when browsers omit it

Browsers often send Referer automatically when the current request comes from a link click, an image fetch, a script fetch, or an XHR or Fetch call from a page. Browsers also omit or reduce Referer in a few important cases:

- Direct navigation from the URL bar or a bookmark does not have a previous page to report.
- HTTPS to HTTP transitions can drop Referer because the destination is less secure.
- Links with `rel="noreferrer"` suppress the header completely.
- A page can tighten the behavior with `Referrer-Policy: no-referrer` or another stricter policy.

Modern browsers also changed the default behavior. MDN documents `strict-origin-when-cross-origin` as the current default. Same-origin requests can include the full path. Cross-origin requests usually send only the origin.

That detail matters for scraping. If you send a full product URL on a cross-origin API call, you create an anomaly. A browser would usually send only the origin. Referer is not about having a value alone. Referer is about sending the right amount of detail.



## How Websites Use the Referer Header to Detect Scrapers

Many scraping guides say "set Referer to google.com" and move on. That advice is weak because the server is not checking the header in isolation. The server is checking whether the requested URL, the session state, and the Referer value fit together.

### Pattern 1: Missing Referer on deep pages

This is the most common failure mode. Most HTTP clients do not invent a navigation path for you, so a request to a deep page or internal API often leaves Referer blank.

Real users do reach deep URLs directly sometimes. But large volumes of direct requests to `/product/123`, `/checkout`, or `/api/reviews` with no Referer can still look unnatural. A website expects many of those requests to come from category pages, product pages, or UI interactions inside the same session.

### Pattern 2: Implausible Referer values

A wrong value can be worse than a missing value. If you set `Referer: https://www.google.com/` for an internal API that is only called from `https://example.com/product/123`, the server now sees an impossible path.

This is why a generic search engine Referer is rarely the right fix. Search engines might make sense for the first landing request to a public page. Search engines usually do not make sense for JSON APIs, background requests, cart endpoints, or review widgets.

### Pattern 3: Broken Referer chains

A real session usually builds a believable path:

1. The first page may have no Referer at all.
2. The next page uses the first page as Referer.
3. The next API call uses the current page as Referer.

Scrapers often break that chain by jumping straight to the last step. The server now sees cookies, path depth, and Referer values that do not line up.

The result is not always an explicit block. Sometimes the result is softer. The request gets lower trust, the challenge score rises, or the API silently returns less data. That is why Referer belongs in the broader [header fingerprinting](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-headers) conversation rather than a one-line workaround.

[How Headers Are Used to Block Web Scrapers and How to Fix ItIntroduction to web scraping headers - what do they mean, how to configure them in web scrapers and how to avoid being blocked.](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-headers)

## Referer-Gated APIs: A Special Case

Some APIs do not use Referer as a soft anti-bot signal. Some APIs use Referer as a hard gate. If the header is missing or wrong, the API returns a validation error instead of data.

The [Scrapfly Referer Scrapeground](https://scrapfly.io/scrapeground/headers/referer) shows a clean example with [web-scraping.dev/testimonials](https://web-scraping.dev/testimonials). If you request `https://web-scraping.dev/api/testimonials` without a Referer header, the API returns `422 Unprocessable Entity`. If you send `Referer: https://web-scraping.dev/testimonials`, the same endpoint returns `200 OK`.

python```python
import httpx

API_URL = "https://web-scraping.dev/api/testimonials"
REFERER_URL = "https://web-scraping.dev/testimonials"

missing = httpx.get(API_URL, timeout=30.0)
print(missing.status_code)
print(missing.text)

fixed = httpx.get(API_URL, headers={"Referer": REFERER_URL}, timeout=30.0)
print(fixed.status_code)
```



Running the script shows the difference clearly:

text```text
422
{"detail":[{"type":"missing","loc":["header","referer"],"msg":"Field required","input":null}]}
200
```



The first request fails with a `422` and a JSON error body because the Referer header is missing. The second request returns `200` because the correct Referer is present. This proves two things. First, not every Referer problem is "anti-bot" in the fuzzy scoring sense. Second, the right fix comes from browser devtools, not guesswork. Open the Network tab, inspect the successful browser request, and copy the exact Referer value the page sent.

You can save time with a short workflow:

1. Open the page that triggers the API call in your browser.
2. Inspect the API request in the Network panel and copy the Referer value.
3. Reproduce that exact value in your scraper before you start tuning anything else.



Scrapfly

#### Scale your web scraping effortlessly

Scrapfly handles proxies, browsers, and anti-bot bypass — so you can focus on data.

[Try Free →](https://scrapfly.io/register)## How to Set the Referer Header When Web Scraping

The general rule is simple: set Referer to the page a real browser would have been on right before the current request. For landing pages, that may be a search results page or a category page. For internal APIs, that is usually the page containing the widget or script that triggers the call.

### Python requests and httpx

For one-off requests, both [requests](https://scrapfly.io/blog/posts/python-requests-headers-guide) and [httpx](https://scrapfly.io/blog/posts/web-scraping-with-python-httpx) let you pass a `headers` dictionary directly. For multi-step scraping, you should think in terms of a session or client and update Referer as the path changes.

python```python
import httpx
import requests

HOME_URL = "https://web-scraping.dev/"
PAGE_URL = "https://web-scraping.dev/testimonials"
API_URL = "https://web-scraping.dev/api/testimonials"

with requests.Session() as session:
    session.headers.update({"User-Agent": "Mozilla/5.0"})
    session.get(HOME_URL, timeout=30)
    session.get(PAGE_URL, headers={"Referer": HOME_URL}, timeout=30)
    api_response = session.get(API_URL, headers={"Referer": PAGE_URL}, timeout=30)
    print(api_response.status_code)

with httpx.Client(headers={"User-Agent": "Mozilla/5.0"}, timeout=30.0) as client:
    client.get(HOME_URL)
    client.get(PAGE_URL, headers={"Referer": HOME_URL})
    api_response = client.get(API_URL, headers={"Referer": PAGE_URL})
    print(api_response.status_code)
```



Both sessions print `200` because the Referer chain matches a real navigation path:

text```text
200
200
```



This pattern is small, but it fixes the most common mistake. You are no longer sending the same fake Referer to every request. You are building a believable chain that matches how the session moved from the homepage to the HTML page and then to the API.

If you also need a realistic `User-Agent`, cookies, or browser-like request sequencing, pair this approach with the techniques in [our headers guide](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-headers). Referer works best when the rest of the request looks believable too.

### Scrapy and RefererMiddleware

[Scrapy](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) already ships with `RefererMiddleware`. Many normal `response.follow()` flows produce natural Referer values without extra work. That default helps when your spider follows links the same way a browser would.

You still need manual control for referer-gated APIs or unusual navigation. In those cases, set the header directly on the `Request`, and only disable the middleware if it gets in your way.

python```python
import json
import scrapy


class TestimonialsSpider(scrapy.Spider):
    name = "testimonials_referer"
    start_urls = ["https://web-scraping.dev/testimonials"]

    def parse(self, response):
        yield scrapy.Request(
            url="https://web-scraping.dev/api/testimonials",
            headers={"Referer": response.url},
            callback=self.parse_api,
        )

    def parse_api(self, response):
        data = json.loads(response.text)
        yield {"count": len(data)}
```



Running the spider logs one item with the testimonial count:

text```text
{'count': 5}
```



The spider above uses the HTML page URL as the Referer for the API call. That is exactly what the server expects for this target. If your project needs stricter behavior across the whole crawl, Scrapy also exposes `REFERRER_POLICY` and per-request `referrer_policy` controls. Use those when policy matters. Use explicit `headers={"Referer": ...}` when one endpoint needs a known value.

[Web Scraping With Scrapy: The Complete Guide in 2026Tutorial on web scraping with scrapy and Python through a real world example project. Best practices, extension highlights and common challenges.](https://scrapfly.io/blog/posts/web-scraping-with-scrapy)

### Playwright and Puppeteer

Browser automation usually gets Referer right when you let the page move through the site naturally. A click on a link, a form submit, or a fetch call from the right document already carries the right context.

Manual Referer work starts when you skip the page and jump straight to the endpoint. [Playwright](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python) and [Puppeteer](https://scrapfly.io/blog/posts/web-scraping-with-puppeteer-and-nodejs) both let you set extra headers for those cases.

python```python
import httpx
import requests

HOME_URL = "https://web-scraping.dev/"
PAGE_URL = "https://web-scraping.dev/testimonials"
API_URL = "https://web-scraping.dev/api/testimonials"

with requests.Session() as session:
    session.headers.update({"User-Agent": "Mozilla/5.0"})
    session.get(HOME_URL, timeout=30)
    session.get(PAGE_URL, headers={"Referer": HOME_URL}, timeout=30)
    api_response = session.get(API_URL, headers={"Referer": PAGE_URL}, timeout=30)
    print(api_response.status_code)

with httpx.Client(headers={"User-Agent": "Mozilla/5.0"}, timeout=30.0) as client:
    client.get(HOME_URL)
    client.get(PAGE_URL, headers={"Referer": HOME_URL})
    api_response = client.get(API_URL, headers={"Referer": PAGE_URL})
    print(api_response.status_code)
```



The script prints `200` because the Referer matches what the server expects:

text```text
200
```



This Playwright example shows the two knobs that matter most: `context.set_extra_http_headers()` for repeated requests and `page.goto(..., referer=...)` for a single request. If you only need to modify specific URL patterns, use `route.continue_()` and inject the header there instead of changing the whole context.

javascript```javascript
import puppeteer from "puppeteer";

const PAGE_URL = "https://web-scraping.dev/testimonials";
const API_URL = "https://web-scraping.dev/api/testimonials";

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setExtraHTTPHeaders({ referer: PAGE_URL });
await page.goto(API_URL);
await browser.close();
```



The Puppeteer example uses `page.setExtraHTTPHeaders()` for the same reason. One warning from the official Puppeteer docs matters here: `setExtraHTTPHeaders()` does not guarantee outgoing header order. If header order also matters on your target, treat Referer as only one piece of the fingerprint.

[Web Scraping with Playwright and PythonPlaywright is the new, big browser automation toolkit - can it be used for web scraping? In this introduction article, we'll take a look how can we use Playwright and Python to scrape dynamic websites.](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python)

## Setting Referer with Scrapfly



ScrapFly provides [web scraping](https://scrapfly.io/docs/scrape-api/getting-started), [screenshot](https://scrapfly.io/docs/screenshot-api/getting-started), and [extraction](https://scrapfly.io/docs/extraction-api/getting-started) APIs for data collection at scale.

- [Anti-bot protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - scrape web pages without blocking!
- [Rotating residential proxies](https://scrapfly.io/docs/scrape-api/proxy) - prevent IP address and geographic blocks.
- [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - scrape dynamic web pages through cloud browsers.
- [Full browser automation](https://scrapfly.io/docs/scrape-api/javascript-scenario) - control browsers to scroll, input and click on objects.
- [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - scrape as HTML, JSON, Text, or Markdown.
- [Full screenshot customization](https://scrapfly.io/docs/screenshot-api/getting-started#api_param_capture) - scroll and capture exact areas.
- [Comprehensive options](https://scrapfly.io/docs/screenshot-api/getting-started) - block banners, use dark mode, and more.
- [LLM prompts](https://scrapfly.io/docs/extraction-api/llm-prompt) - extract data or ask questions using LLMs
- [Extraction models](https://scrapfly.io/docs/extraction-api/automatic-ai) - automatically find objects like products, articles, jobs, and more.
- [Extraction templates](https://scrapfly.io/docs/extraction-api/rules-and-template) - extract data using your own specification.
- [Python](https://scrapfly.io/docs/sdk/python) and [Typescript](https://scrapfly.io/docs/sdk/typescript) SDKs, as well as [Scrapy](https://scrapfly.io/docs/sdk/scrapy) and [no-code tool integrations](https://scrapfly.io/docs/integration/getting-started).

Scrapfly supports custom headers through `ScrapeConfig(headers={...})`, so referer-gated APIs are straightforward. That is the right choice when you already know the exact Referer value from browser devtools or from the Scrapeground example.

python```python
from scrapfly import ScrapflyClient, ScrapeApiResponse, ScrapeConfig

API_URL = "https://web-scraping.dev/api/testimonials"
REFERER_URL = "https://web-scraping.dev/testimonials"

client = ScrapflyClient(key="YOUR_SCRAPFLY_KEY")

api_response: ScrapeApiResponse = client.scrape(
    ScrapeConfig(
        url=API_URL,
        headers={"Referer": REFERER_URL},
    )
)
print(api_response.upstream_status_code)

api_response = client.scrape(
    ScrapeConfig(
        url=API_URL,
        asp=True,
        headers={"Referer": REFERER_URL},
    )
)
print(api_response.upstream_status_code)
```



The first request sets the header manually. The second request keeps the manual Referer but also enables `asp=True`. Scrapfly's docs explain that ASP can tune Referer automatically when you do not pass one, but referer-gated APIs are different. If the target expects one exact page URL, you should still send that exact value yourself.

javascript```javascript
import { ScrapflyClient, ScrapeConfig } from "scrapfly-sdk";

const client = new ScrapflyClient({ key: process.env.SCRAPFLY_KEY });

const result = await client.scrape(new ScrapeConfig({
  url: "https://web-scraping.dev/api/testimonials",
  headers: { Referer: "https://web-scraping.dev/testimonials" },
  asp: true,
}));

console.log(result.result.status_code);
```



The JavaScript SDK works the same way. Pass the header when the target requires one exact Referer, and use ASP for the broader anti-bot work around the request.

### Web Scraping API

Scrape any website with our powerful API. Anti-bot bypass, JavaScript rendering, and rotating proxies built-in.



[Try Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started)





## FAQ

What should I set as the Referer header for web scraping?Set Referer to the page a real browser would have been on right before the current request. For a product page, that might be the category or search page. For an internal API, that is usually the HTML page that triggers the request. Avoid generic values unless the request starts from a search engine.







Is the Referer header always sent?No. Browsers can omit Referer on direct navigation, bookmarks, `rel="noreferrer"` links, stricter referrer policies, and some secure-to-insecure transitions. Missing Referer is not automatically a bot signal. The real problem is often a Referer value that does not match the navigation path the server expects.







What is the difference between Referer and Origin?`Origin` only carries scheme, host, and port. `Referer` can carry the full URL path as well, depending on policy. Servers use `Origin` heavily for CORS and CSRF checks. Servers use `Referer` more for navigation context, analytics, and lightweight access control. Some endpoints check both, so copy them from a real browser together when needed.







Should I always set Referer to google.com when web scraping?No. `google.com` can be a believable source for the first landing request to a public page, but it is usually wrong for product APIs, review APIs, or AJAX endpoints. Those requests normally come from the current site, not from a search engine. A wrong Referer is often more suspicious than leaving the header out.







Why does my scraper still get blocked even with the correct Referer?Because Referer is only one signal. Websites also check cookies, TLS fingerprints, `User-Agent`, header order, browser APIs, IP reputation, and request timing. A correct Referer removes one obvious mismatch, but it cannot save a request that still looks synthetic everywhere else. Treat Referer as necessary context, not a complete bypass.









## Summary

The Referer header works when it reflects a real path through the site. Missing Referer can look odd on deep endpoints. A fake search-engine Referer can look even worse on internal APIs. The safest workflow is simple. Inspect the successful browser request, copy the Referer value the browser used, and then reproduce that value in your client, spider, or browser automation flow.

For broader request realism, pair this article with [our full guide to scraping headers](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-headers). If you need a live target to test against, use [Scrapfly's Referer Scrapeground](https://scrapfly.io/scrapeground/headers/referer) and compare the failed request against the fixed one.

Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect:

- Do not scrape at rates that could damage the website.
- Do not scrape data that's not available publicly.
- Do not store PII of EU citizens protected by GDPR.
- Do not repurpose *entire* public datasets which can be illegal in some countries.

Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer.



 

   Table of Contents















 

  Table of Contents- [Key Takeaways](#key-takeaways)
- [What Is the HTTP Referer Header?](#what-is-the-http-referer-header)
- [Why the header is misspelled](#why-the-header-is-misspelled)
- [When browsers send Referer and when browsers omit it](#when-browsers-send-referer-and-when-browsers-omit-it)
- [How Websites Use the Referer Header to Detect Scrapers](#how-websites-use-the-referer-header-to-detect-scrapers)
- [Pattern 1: Missing Referer on deep pages](#pattern-1-missing-referer-on-deep-pages)
- [Pattern 2: Implausible Referer values](#pattern-2-implausible-referer-values)
- [Pattern 3: Broken Referer chains](#pattern-3-broken-referer-chains)
- [Referer-Gated APIs: A Special Case](#referer-gated-apis-a-special-case)
- [How to Set the Referer Header When Web Scraping](#how-to-set-the-referer-header-when-web-scraping)
- [Python requests and httpx](#python-requests-and-httpx)
- [Scrapy and RefererMiddleware](#scrapy-and-referermiddleware)
- [Playwright and Puppeteer](#playwright-and-puppeteer)
- [Setting Referer with Scrapfly](#setting-referer-with-scrapfly)
- [Web Scraping API](#web-scraping-api)
- [FAQ](#faq)
- [Summary](#summary)
 
    Join the Newsletter  Get monthly web scraping insights 

 

  



Scale Your Web Scraping

Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits.

  No credit card required  1,000 free API credits  Anti-bot bypass included 

 [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) 

 Not ready? Get our newsletter instead. 

 

## Explore this Article with AI

 [ ChatGPT ](https://chat.openai.com/?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhttp-referer-header-complete-guide-for-web-scraping) [ Gemini ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhttp-referer-header-complete-guide-for-web-scraping) [ Grok ](https://x.com/i/grok?text=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhttp-referer-header-complete-guide-for-web-scraping) [ Perplexity ](https://www.perplexity.ai/search/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhttp-referer-header-complete-guide-for-web-scraping) [ Claude ](https://claude.ai/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhttp-referer-header-complete-guide-for-web-scraping) 



 ## Related Articles

 [  

 http nodejs 

### How to Set Axios Headers: Complete Guide with Examples (2026)

Learn how to set, manage, and troubleshoot Axios headers: per-request config, global defaults, instances, interceptors, ...

 

 ](https://scrapfly.io/blog/posts/guide-to-javascript-axios-headers) [  

 http blocking 

### How Headers Are Used to Block Web Scrapers and How to Fix It

Introduction to web scraping headers - what do they mean, how to configure them in web scrapers and how to avoid being b...

 

 ](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-headers) [     

 blocking 

### 5 Tools to Scrape Without Blocking and How it All Works

Tutorial on how to avoid web scraper blocking. What is javascript and TLS (JA3) fingerprinting and what role request hea...

 

 ](https://scrapfly.io/blog/posts/how-to-scrape-without-getting-blocked-tutorial) 

  



   



 Scale your web scraping effortlessly, **1,000 free credits** [Start Free](https://scrapfly.io/register)