Webhook

Introduction

Webhook allow you make asynchronous scrape. When calling api with webhook_name=__your_webhook_name__ your call is queue and as soon as processed by our system, when scrape is done we notify you through the defined webhook as you configured it.

You can managed webhook through the web interface

You are able to configure Content-Type and Content-Encoding to optimize transfer. By default no compression and json serialization is used.

The body sent to your endpoint is exactly the same as regular api scrape response plus webhook information in the context part.

See in your dashboard

Scope

Webhooks are scoped per project per environment. Don't be angry if the API reject when you pass webhook_name, the webhook is not in the same project / environment as your API key.

Retry Policy

If we can't notify the endpoint specified in your webhook settings.

  • 30 seconds
  • 1 minute
  • 5 minutes
  • 30 minutes
  • 1 hour
  • 1 day
If we failed to reach your application more than 100 in a row, the system automatically disable it and you will be notified. You can re enable it from the UI at any moment.

Development

Useful tools to develop locally :

Security

For security reason, you can set define a secret, Scrapfly will reach you with X-Scrapfly-Webhook-Secret header containing the defined secret

Headers

Following headers are added :

  • X-Scrapfly-Webhook-Env : Related environment where webhook is triggered
  • X-Scrapfly-Webhook-Project : Related project name
  • X-Scrapfly-Webhook-Secret Related Secret to authenticate origin of call
  • X-Scrapfly-Webhook-Name Name of webhook

API

To enable webook, you must pass webhook_name parameter to API

You must create and configure webhook named example in order to play this example.
Interactive Example: API Webhook Example
openapi openapi API Webhook Example Sign in
            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&webhook_name=sdfsdfsdf
        
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&webhook_name=sdfsdfsdf

key
=
url
= https%3A%2F%2Fhttpbin.org%2Fanything
webhook_name
= sdfsdfsdf
openapi openapi API Webhook Result | Json
            {
    ...
    "context": {
        ...
        "webhook": {
            "name": "example",
            "secret": "1a8c1001-5bc9-4b23-a830-31ac864cd18c",
            "consecutive_failed_count": 0
        }
        ...
    }
    ...
}