# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Unblocker (formerly ASP)](https://scrapfly.io/docs/scrape-api/unblocker) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Batch (Multi-URL Scraping)](https://scrapfly.io/docs/scrape-api/batch) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Schedule](https://scrapfly.io/docs/scrape-api/schedule) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Search](https://scrapfly.io/docs/crawler-api/search) - [Prompt & Extract](https://scrapfly.io/docs/crawler-api/prompt) - [Auto Refresh](https://scrapfly.io/docs/crawler-api/refresh) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Schedule](https://scrapfly.io/docs/crawler-api/schedule) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Schedule](https://scrapfly.io/docs/screenshot-api/schedule) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Data API #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [Captcha Solver](https://scrapfly.io/docs/cloud-browser-api/captcha-solver) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) - [Native Browser MCP](https://scrapfly.io/docs/cloud-browser-api/mcp) - [DevTools Protocol](https://scrapfly.io/docs/cloud-browser-api/cdp-reference) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Webhook Payload Reference When an alert transitions to TRIGGERED (or resolves), Scrapfly sends an HTTP POST to every webhook URL you have configured on that alert. This page documents the exact JSON payload, the headers sent with each delivery, and how to verify the HMAC signature. ## JSON payload Every delivery sends a JSON body with this shape: ``` { "event_id": "01J0AB12CDEFGH3JKLMNPQRS56", "alert_uuid": "01HXYZ123456789ABCDEF01234", "project_uuid": "01HPQR987654321ZYXWVU01234", "name": "Block rate > 20% on web-scraping.dev", "transition": "pending_to_triggered", "fired_at": "2026-05-17T12:34:56Z", "metric": { "id": "scrape.block_rate", "dimensions": { "root_domain": "web-scraping.dev" }, "window_m": 5, "value": 34.5, "threshold": 20.0, "comparator": "gt" }, "links": { "dashboard": "https://scrapfly.io/dashboard/alerts/01HXYZ123456789ABCDEF01234", "graph": "https://scrapfly.io/dashboard/alerts/01HXYZ123456789ABCDEF01234#alert-graph" } } ``` ### Field reference | Field | Type | Description | |---|---|---| | `event_id` | string (ULID) | Stable per-event identifier. Use this as your deduplication key. Scrapfly does not retry a failed delivery. Also mirrored in the `X-Scrapfly-Webhook-Id` header. | | `alert_uuid` | string (ULID) | Unique identifier for the alert definition | | `project_uuid` | string (ULID) | Project the alert belongs to | | `name` | string | Human-readable alert name you set at creation time | | `transition` | string | State change that triggered this notification. Transition labels are built as `_to_` (lowercase), plus two special values. These are the only values a delivery can carry: - `pending_to_triggered`: alert just fired. A first breach always lands in PENDING first, so this, not `ok_to_triggered`, is the label on the first notification - `triggered_to_ok`: alert resolved (recovery\_minutes = 0) - `recovering_to_ok`: alert resolved after the recovery window elapsed - `re_notify`: still triggered, re-notify interval elapsed - `test_fire`: synthetic notification from the test endpoint The state machine has transitions that deliberately notify nobody, TRIGGERED to RECOVERING and RECOVERING back to TRIGGERED among them, so their labels never reach a webhook. See [State machine](https://scrapfly.io/docs/alerting/state-machine) for the complete transition table. | | `fired_at` | string (RFC 3339) | Timestamp when this notification was generated | | `metric.id` | string | Metric ID from the [metric registry](https://scrapfly.io/docs/alerting/metric-reference) | | `metric.dimensions` | object | Dimension filter key/value pairs applied to this alert (may be empty) | | `metric.window_m` | integer | Evaluation window in minutes used for this observation | | `metric.value` | float | Raw metric value at the time the notification was triggered | | `metric.threshold` | float | Threshold you configured on the alert | | `metric.comparator` | string enum | `gt`, `lt`, `gte`, `lte`, `eq`, `neq` | | `links.dashboard` | string (URL) | Direct link to the alert detail page in the dashboard | | `links.graph` | string (URL) | Direct link to the metric graph for this alert | ## HTTP headers Every alert webhook delivery includes these headers: | Header | Example value | Description | |---|---|---| | `Content-Type` | `application/json` | Always JSON for alert webhooks | | `User-Agent` | `Scrapfly/Webhook` | Identify Scrapfly in your server logs | | `X-Scrapfly-Webhook-Timestamp` | `1747531296` | Unix-seconds timestamp at which Scrapfly signed the request. Included in the signed string (see verification below) so you can reject stale deliveries for replay protection. | | `X-Scrapfly-Webhook-Signature` | `3A9F2C...D8E1` | HMAC-SHA256 of `"."`, encoded as uppercase hex. Key is the UTF-8 encoding of your signing secret. See the verification section for the exact algorithm. | | `X-Scrapfly-Webhook-Signature-Lowercase` | `3a9f2c...d8e1` | Same digest, lowercase. Provided for runtimes and proxies that normalize header values to lowercase. | | `X-Scrapfly-Webhook-Resource-Type` | `alert` | Use this header to distinguish alert deliveries from scrape webhook deliveries if a single endpoint receives both. Alert deliveries always set this to `alert`. | | `X-Scrapfly-Webhook-Id` | `01J0AB...` | Mirrors the JSON body's `event_id`. Use it as your deduplication key. NOT the alert\_uuid. | **Existing scrape webhook users:** If your endpoint already verifies the `X-Scrapfly-Webhook-Signature` header for scrape callbacks, the HMAC algorithm and key encoding are identical. The only difference is the header `X-Scrapfly-Webhook-Resource-Type` changes from `scrape` to `alert`, and the JSON body shape is different. ## Verifying the signature Scrapfly signs each delivery with HMAC-SHA256 over a string that combines the timestamp header and the raw request body. Including the timestamp in the signature lets you reject stale or replayed requests. The signing key is the UTF-8 encoding of the alert's HMAC signing secret. Scrapfly generates that secret when the alert is created; it cannot be supplied by the caller. Read it from the `hmac_key` field of `GET /alert/:alert_uuid`. The signed message is `"."` (literal dot between them). Do not parse and re-serialize the JSON body before signing; that changes the byte sequence. Algorithm summary: 1. Read the raw request body (before JSON parsing). 2. Read `X-Scrapfly-Webhook-Timestamp` (Unix seconds). 3. Optionally reject the delivery if the timestamp is more than a few minutes from your current clock (replay protection). 4. Compute `HMAC-SHA256(key=secret.encode('utf-8'), msg=(timestamp + "." + raw_body).encode())`. 5. Hex-encode the digest (uppercase). 6. Use a **constant-time comparison** to compare with the value in `X-Scrapfly-Webhook-Signature`. Python cURL (test delivery) Flask example. The same pattern works with FastAPI, Django, or any WSGI/ASGI framework that exposes raw request bytes. ``` import hmac import hashlib import time from flask import Flask, request, abort app = Flask(__name__) SIGNING_SECRET = "your-webhook-signing-secret" # hmac_key from GET /alert/:alert_uuid MAX_SKEW_SECONDS = 300 # reject deliveries older than 5 minutes @app.route("/scrapfly/alert", methods=["POST"]) def receive_alert(): # 1. Read raw bytes BEFORE any JSON parsing raw_body = request.get_data() # 2. Replay protection: reject stale deliveries ts = request.headers.get("X-Scrapfly-Webhook-Timestamp", "") if not ts.isdigit() or abs(int(time.time()) - int(ts)) > MAX_SKEW_SECONDS: abort(401, "Stale or missing timestamp") # 3. Compute HMAC-SHA256 over "." signed_message = ts.encode("utf-8") + b"." + raw_body computed = hmac.new( SIGNING_SECRET.encode("utf-8"), signed_message, hashlib.sha256, ).hexdigest().upper() # 4. Constant-time comparison (prevents timing attacks) received = request.headers.get("X-Scrapfly-Webhook-Signature", "") if not hmac.compare_digest(computed, received): abort(401, "Signature mismatch") # 5. Parse and handle the payload payload = request.get_json() resource_type = request.headers.get("X-Scrapfly-Webhook-Resource-Type", "") if resource_type == "alert": transition = payload["transition"] alert_name = payload["name"] metric_value = payload["metric"]["value"] print(f"Alert [{transition}]: {alert_name} = {metric_value}") return "", 200 ``` Use the test-fire endpoint to send a synthetic notification through all channels configured on an alert without affecting state: ``` # Replace ALERT_UUID with the UUID from the alert detail page # and with your scp-live-... project key. curl -X POST "https://api.scrapfly.io/alert/ALERT_UUID/test?key=" \ -H "Content-Type: application/json" ``` The test delivery sends the same headers and signing scheme as a real notification, with `transition` set to `test_fire`. This lets you verify HMAC signing end-to-end without waiting for a real alert to fire. ## Deduplication Alert webhook deliveries are fire-and-forget. Each event is POSTed once, bounded by a hard 5-second timeout, and is **not** retried on timeout or a 5xx response: a failed delivery is recorded on the alert's event log and dropped. Use `event_id` as your deduplication key if anything on your own side (a proxy, queue, or load balancer) can replay a request: ``` dedup_key = payload["event_id"] # or equivalently: dedup_key = request.headers["X-Scrapfly-Webhook-Id"] ``` Do NOT dedup on `alert_uuid` alone: a single alert fires many distinct events over its lifetime (pending\_to\_triggered, re\_notify after each interval, triggered\_to\_ok) and they all share the same alert\_uuid but each has a unique event\_id. ## Routing multi-product webhook endpoints A single webhook URL can receive deliveries from multiple Scrapfly products (scrape callbacks, alert notifications, crawler callbacks, etc.). Use the `X-Scrapfly-Webhook-Resource-Type` header to route: ``` resource_type = request.headers.get("X-Scrapfly-Webhook-Resource-Type", "") if resource_type == "scrape": handle_scrape_callback(payload) elif resource_type == "alert": handle_alert_notification(payload) elif resource_type == "crawl": handle_crawler_callback(payload) else: # Unknown type: log and ignore for forward compatibility pass ``` [ Metric Reference ](https://scrapfly.io/docs/alerting/metric-reference) [ Anti-Spam Controls ](https://scrapfly.io/docs/alerting/anti-spam)