Webhook

View as markdown

Copy for LLM

Copy for LLM Open in ChatGPT Open in Claude Open in Perplexity

Scrapfly's webhook feature is ideal for managing screenshot tasks asynchronously. When webhook is specified through the webhook_name parameter, Scrapfly will call your HTTP endpoint with the screenshot response.

To start using webhooks first one must be created using webhook web interface.

The body sent to your endpoint is the same as a regular Screenshot API response. For reconciliation, you will get the job_uuid and webhook_uuid in the response headers.

Webhook Queue Size

The webhook queue size indicates the maximum number of queued webhooks that can be scheduled. After the scraping process is completed and your application is notified, the queue size is reduced. This allows you to schedule additional screenshots beyond the concurrency limit of your subscription. The scheduler will handle this and ensure that your concurrency limit is met.

FREE $0.00/mo	DISCOVERY $30.00/mo	PRO $100.00/mo	STARTUP $250.00/mo	ENTERPRISE $500.00/mo
500	500	2,000	5,000	10,000

See in Your Dashboard

Scope

Webhooks are scoped per scrapfly projects and environments. Make sure to create a webhook for each of your project and environment (test/live).

Usage

Webhook can be used for multiple purpose, in context of the Screenshot API, to assert you received a screenshot, you must check the header X-Scrapfly-Webhook-Resource-Type and check the value is screenshot

To enable webhook callbacks, all you need to do is specify the webhook_name parameter in your screenshot requests. Then, Scrapfly will immediately return a promise response and call your webhook endpoint as soon as the screenshot is done.

Note that your webhook has to be configured to respond to 2xx response code for webhook to be considered a success. The 3xx redirect responses will be followed and response codes 4xx and 5xx are considered failures and will be retried as per the retry policy.

The below examples assume you have a webhook named my-webhook registered. You can create a webhook named "example" via the web dashboard.

curl -G \
--request "GET" \
--url "https://api.scrapfly.io/screenshot" \
--data-urlencode "" \
--data-urlencode "webhook_name=my-webhook" \
--data-urlencode "url=https://web-scraping.dev/product/1"

Example Of Response

{
  "job_uuid": "a0e6f3e8-be35-438a-942a-be77aa545d30",
  "log_uuid": "a0e6f3e8-be35-438a-942a-be77aa545d30",
  "log_url": "https://api.scrapfly.io/scrape/log/a0e6f3e8-be35-438a-942a-be77aa545d30",
  "success": true,
  "webhook_enqueued": true,
  "webhook_name": "my-webhook",
  "webhook_queue_limit": 10000,
  "webhook_queued_element": 1,
  "webhook_uuid": "cdf37252-fea7-4267-a568-aa0e5964ee21"
}

Tracking

When you enqueue a screenshot request, you receive:

• job_uuid - A unique job identifier for reconciliation
• log_uuid - The log identifier for debugging and monitoring
• log_url - Direct link to the request log in Scrapfly dashboard

When your webhook is notified, you will retrieve the processed job id in the response header X-Scrapfly-Webhook-Job-Id to reconcile it in your system. The log headers X-Scrapfly-Log-Uuid and X-Scrapfly-Log-Url are also included in the enqueue response for immediate access to the request log.

Retry Policy

Webhook callbacks are retried if Scrapfly can't notify the endpoint specified in your webhook settings based on this retry policy:

• 30 seconds
• 1 minute
• 5 minutes
• 30 minutes
• 1 hour
• 1 day

If we failed to reach your application more than 100 times in a row, the system automatically disables it, and you will be notified. You can re-enable it from the UI at any point after.

Development

Useful tools to develop locally :

• https://webhook.site Collect and display webhook
• https://ngrok.com Expose you local application through a secured tunnel to the internet

Security

Webhooks are signed using HMAC (Hash-based Message Authentication Code) with the SHA-256 algorithm to ensure the integrity of the webhook content and verify its authenticity. This mechanism helps prevent tampering and ensures that webhook payloads are from trusted sources.

HMAC Overview

HMAC is a cryptographic technique that combines a secret key with a hash function (in this case, SHA-256) to produce a fixed-size hash value known as the HMAC digest. This digest is unique to both the original message and the secret key, providing a secure way to verify the integrity and authenticity of the message.

Signature in HTTP Header

When sending a webhook request, a signature is generated using HMAC-SHA256 and included in the HTTP header X-Scrapfly-Webhook-Signature. This signature is computed based on the webhook payload and a secret key known only to the sender and receiver.

Verification Process

Upon receiving a webhook request, the receiver extracts the payload and computes its own HMAC-SHA256 signature using the same secret key. It then compares this computed signature with the signature provided in the X-Scrapfly-Webhook-Signature header. If the two signatures match, it indicates that the payload has not been tampered with and originates from the expected sender.

import hmac
import hashlib

# Example secret key (replace with actual secret key)
secret_key = b'my_secret_key'

# Example webhook payload (replace with actual payload)
webhook_payload = b'{"data": "example"}'

# Compute HMAC-SHA256 signature
computed_signature = hmac.new(secret_key, webhook_payload, hashlib.sha256).hexdigest()

# Compare computed signature with received signature
received_signature = '...'  # Extracted from X-Scrapfly-Webhook-Signature header
if computed_signature == received_signature:
    print("Signature verification successful. Payload is authentic.")
else:
    print("Signature verification failed. Payload may have been tampered with.")

Security Considerations

• Keep Secret Key Secure: The secret key used for HMAC computation should be kept confidential and not exposed publicly.
• Use HTTPS: Webhook communication should be conducted over HTTPS to ensure data privacy and integrity during transit.
• Regular Key Rotation: Periodically rotate the secret key used for HMAC computation to enhance security.

Headers

Enqueue Response Headers

When you enqueue a screenshot request with a webhook, the following headers are returned:

• X-Scrapfly-Log-Uuid : Unique log identifier for the request
• X-Scrapfly-Log-Url : Direct URL to view the request log in the dashboard

Webhook Callback Headers

When Scrapfly calls your webhook endpoint, the following headers are included:

• X-Scrapfly-Webhook-Env : Related environment where webhook is triggered
• X-Scrapfly-Webhook-Project : Related project name
• X-Scrapfly-Webhook-Signature : HMAC SHA-256 Integrity Signature
• X-Scrapfly-Webhook-Name : Name of the webhook
• X-Scrapfly-Webhook-Resource-Type : Resource type (e.g., screenshot)
• X-Scrapfly-Webhook-Job-Id : Unique Job Identifier given in the enqueue call
• X-Scrapfly-Log-Uuid : Unique log identifier for the request
• X-Scrapfly-Log-Url : Direct URL to view the request log in the dashboard

Related Errors

All related errors are listed below. You can see the full description and example of the error response on Errors section of the documentation.

• ERR::WEBHOOK::DISABLED - Given webhook is disabled, please check out your webhook configuration for the current project / env
• ERR::WEBHOOK::ENDPOINT_UNREACHABLE - We were not able to contact your endpoint
• ERR::WEBHOOK::QUEUE_FULL - You reach the maximum concurrency limit
• ERR::WEBHOOK::MAX_RETRY - Maximum retry exceeded on your webhook
• ERR::WEBHOOK::NOT_FOUND - Unable to find the given webhook for the current project / env
• ERR::WEBHOOK::QUEUE_FULL - You reach the limit of scheduled webhook - You must wait pending webhook are processed

Pricing

No additional fee applied on usage.

Integration

• Python SDK Built-in server and example