Webhook allow you make asynchronous scrape. When calling api with
webhook_name=__your_webhook_name__ your call is queue
and as soon as processed by our system, when scrape is done we notify you through the defined webhook as you configured it.
You can managed webhook through the web interface
You are able to configure
Content-Encoding to optimize transfer.
By default no compression and json serialization is used.
The body sent to your endpoint is exactly the same as regular api scrape response plus webhook information in the context part.
Webhooks are scoped per project per environment. Don't be angry if the API reject when you pass
webhook_name, the webhook is not
in the same project / environment as your API key.
If we can't notify the endpoint specified in your webhook settings.
- 30 seconds
- 1 minute
- 5 minutes
- 30 minutes
- 1 hour
- 1 day
If we failed to reach your application more than 100 in a row, the system automatically disable it and you will be notified. You can re enable it from the UI at any moment.
Useful tools to develop locally :
- https://webhook.site Collect and display webhook
- https://ngrok.com Expose you local application through a secured tunnel to internet
For security reason, you can set define a secret, Scrapfly will reach you with
X-Scrapfly-Webhook-Secret header containing
the defined secret
Following headers are added :
X-Scrapfly-Webhook-Env: Related environment where webhook is triggered
X-Scrapfly-Webhook-Project: Related project name
X-Scrapfly-Webhook-SecretRelated Secret to authenticate origin of call
X-Scrapfly-Webhook-NameName of webhook
To enable webook, you must pass webhook_name parameter to API
You must create and configure webhook named example in order to play this example.
Interactive Example: API Webhook Example
curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&webhook_name=sdfsdfsdf