# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Unblocker (formerly ASP)](https://scrapfly.io/docs/scrape-api/unblocker) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Batch (Multi-URL Scraping)](https://scrapfly.io/docs/scrape-api/batch) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Schedule](https://scrapfly.io/docs/scrape-api/schedule) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Search](https://scrapfly.io/docs/crawler-api/search) - [Prompt & Extract](https://scrapfly.io/docs/crawler-api/prompt) - [Auto Refresh](https://scrapfly.io/docs/crawler-api/refresh) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Schedule](https://scrapfly.io/docs/crawler-api/schedule) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Schedule](https://scrapfly.io/docs/screenshot-api/schedule) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Data API #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [Captcha Solver](https://scrapfly.io/docs/cloud-browser-api/captcha-solver) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) - [Credential Vault](https://scrapfly.io/docs/cloud-browser-api/credential-vault) - [Native Browser MCP](https://scrapfly.io/docs/cloud-browser-api/mcp) - [DevTools Protocol](https://scrapfly.io/docs/cloud-browser-api/cdp-reference) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [1Password](https://scrapfly.io/docs/cloud-browser-api/1password) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Classify API The **Classify API** runs the same anti-bot detection pipeline used by every live Scrapfly scrape against an HTTP response you already have. You pass in a URL, status code, headers and body; it returns whether the target blocked the request and the name of the anti-bot product that matched (Cloudflare, DataDome, PerimeterX, Akamai, Kasada, Imperva, AWS WAF, F5 Shape, and more). Use it when you've fetched a page through your own infrastructure (custom proxy, cached scrape, third-party system) and want a continuously-maintained verdict on whether it's a real response or a block page, without paying for a new scrape. ## When to use Classify - **Verify a response from your own fetch**: paste the response you got back into `/classify` and let Scrapfly decide if it's blocked. Faster and cheaper than re-scraping just to check. - **Decide whether to retry with Scrapfly's Unblocker**: if Classify flags the response as blocked, enable `unblocker=true` on the next call to `/scrape`, or escalate to `browser_unblock`. - **Audit cached or archived pages**: spot block pages that snuck into your cache so you can refresh them cleanly. - **Power local tooling**: the Scrapfly MCP server's `check_if_blocked` tool uses this endpoint under the hood, so AI agents get an authoritative answer instead of a stale local heuristic. ## Request ### Endpoint `POST /classify?key={API_KEY}` ### Body A JSON object describing a single HTTP response: - `url` (string, **required**): the final URL the response came from. - `status_code` (integer, **required**): the HTTP status code (100-599) of the response you're classifying. - `headers` (object, optional): response headers, case-insensitive. Detection relies heavily on signals like `server`, `cf-ray`, `cf-mitigated`, `x-datadome`, `x-akamai-session-info`, etc. Pass everything you have for best accuracy. - `body` (string, optional): the response body as text. HTML challenge pages often carry the most reliable signal (`Just a moment...`, `_cf_chl_opt`, `cd.kasada.io`, etc.). Binary payloads can be passed as an empty string. - `method` (string, optional, default `GET`): the HTTP method the caller used. ``` POST /classify?key={API_KEY} Content-Type: application/json { "url": "https://target.example.com/product/42", "status_code": 403, "headers": { "server": "cloudflare", "cf-ray": "8c0e0a0b0c0d0e0f-DFW", "cf-mitigated": "challenge" }, "body": "Just a moment......", "method": "GET" } ``` ### Headers - `Content-Type: application/json`, required. ### Body size limit The request body is capped at **2 MiB**. Oversize requests are rejected with `ERR::SCRAPE::CLASSIFY_CONFIG` (HTTP 413). If you're classifying unusually large HTML, trim to the first couple hundred KiB. Detection signatures always live near the top of the body. ## Example call - curl - Python - TypeScript - Go - Rust ``` curl -X POST \ 'https://api.scrapfly.io/classify?key={{ YOUR_API_KEY }}' \ -H 'Content-Type: application/json' \ -d '{ "url": "https://target.example.com/", "status_code": 403, "headers": {"server": "cloudflare", "cf-mitigated": "challenge"}, "body": "...Just a moment..." }' ``` ``` import requests from scrapfly import ScrapflyClient client = ScrapflyClient(key="") # Fetch the response yourself (or load it from your cache/proxy). response = requests.get("https://target.example.com/") result = client.classify( url="https://target.example.com/", status_code=response.status_code, headers=dict(response.headers), body=response.text, ) print(result.blocked) # True / False print(result.antibot) # "cloudflare" | "datadome" | ... | None print(result.cost) # 1 ``` ``` import { ScrapflyClient } from "scrapfly-sdk"; const client = new ScrapflyClient({ key: "" }); // Fetch the response yourself (or load it from your cache/proxy). const upstream = await fetch("https://target.example.com/"); const body = await upstream.text(); const headers: Record = {}; upstream.headers.forEach((value, key) => { headers[key] = value; }); const result = await client.classify({ url: "https://target.example.com/", statusCode: upstream.status, headers, body, }); console.log(result.blocked); // true / false console.log(result.antibot); // "cloudflare" | "datadome" | ... | null console.log(result.cost); // 1 ``` ``` package main import ( "context" "fmt" "io" "log" "net/http" scrapfly "github.com/scrapfly/go-scrapfly" ) func main() { client, err := scrapfly.New("") if err != nil { log.Fatal(err) } // Fetch the response yourself (or load it from your cache/proxy). resp, err := http.Get("https://target.example.com/") if err != nil { log.Fatal(err) } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) headers := map[string]string{} for k, v := range resp.Header { if len(v) > 0 { headers[k] = v[0] } } result, err := client.Classify(context.Background(), &scrapfly.ClassifyRequest{ URL: "https://target.example.com/", StatusCode: resp.StatusCode, Headers: headers, Body: string(body), }) if err != nil { log.Fatal(err) } fmt.Println(result.Blocked) // true / false fmt.Println(result.Antibot) // "cloudflare" | "datadome" | ... | "" fmt.Println(result.Cost) // 1 } ``` ``` use scrapfly_sdk::result::classify::ClassifyRequest; use scrapfly_sdk::Client; use std::collections::HashMap; #[tokio::main] async fn main() -> Result<(), Box> { let client = Client::builder() .api_key("") .build()?; // Fetch the response yourself (or load it from your cache/proxy). let resp = reqwest::get("https://target.example.com/").await?; let status = resp.status().as_u16(); let mut headers: HashMap = HashMap::new(); for (k, v) in resp.headers() { if let Ok(s) = v.to_str() { headers.insert(k.as_str().to_string(), s.to_string()); } } let body = resp.text().await?; let result = client .classify(&ClassifyRequest { url: "https://target.example.com/".into(), status_code: status, headers: Some(headers), body: Some(body), method: None, }) .await?; println!("{}", result.blocked); // true / false println!("{:?}", result.antibot); // Some("cloudflare") | ... | None println!("{}", result.cost); // 1 Ok(()) } ``` ## Response The response is a JSON object describing the verdict. The `blocked` field is the headline answer; the rest provide the evidence. - Blocked response - Clean response (not blocked) ``` HTTP/2 200 Content-Type: application/json X-Scrapfly-Api-Cost: 1 X-Scrapfly-Remaining-Api-Credit: 4999999 { "blocked": true, "antibot": "cloudflare", "cost": 1 } ``` ``` HTTP/2 200 Content-Type: application/json X-Scrapfly-Api-Cost: 1 { "blocked": false, "antibot": null, "cost": 1 } ``` ### Fields - `blocked` (boolean): `true` when Scrapfly detected that the upstream response is an anti-bot block page. - `antibot` (string, nullable): the name of the anti-bot product that matched, when one was detected. Examples: `cloudflare`, `datadome`, `perimeterx`, `akamai`, `kasada`, `imperva`, `aws_waf`, `f5_shape`. `null` when `blocked` is `false`. - `cost` (integer): API credits charged for this call. ## Pricing & Billing Each successful call costs **1 API credit**, regardless of payload size or shield count. The charge is reported in the `X-Scrapfly-Api-Cost` response header and recorded under the Web Scraping API product, the same bucket as `/scrape` calls. Errors (HTTP 4xx/5xx) are **not billed**. Invalid URL, missing `status_code`, oversized body, and backend failures all return without charging. ## Errors All Classify failures return `ERR::SCRAPE::CLASSIFY_CONFIG`. The HTTP status and `message` field describe the specific cause: - HTTP 400: request body is not valid JSON. - HTTP 413: request body exceeds 2 MiB. - HTTP 422: `url` missing, not absolute, or has no host; `status_code` outside 100-599. - HTTP 503: backend temporarily unavailable; safe to retry. ## See also - [Unblocker (formerly ASP)](https://scrapfly.io/docs/scrape-api/unblocker): the same shield pipeline applied proactively during a scrape. - [Web Scraping API Getting Started](https://scrapfly.io/docs/scrape-api/getting-started): the main `/scrape` endpoint you should call when Classify says `blocked: true`. - [Batch (Multi-URL Scraping)](https://scrapfly.io/docs/scrape-api/batch): when you need to scrape many URLs at once.