Webhook allow you to make an asynchronous scrape. When calling API with webhook_name=__your_webhook_name__
, your call is in the queue, and as soon as processed by our system, when the scrape is done, we notify you through the defined webhook as you configured it.
You can manage the webhook through the web interface.
You can configure Content-Type
and Content-Encoding
to optimize transfer.
By default no compression and json serialization is used.
The body sent to your endpoint is precisely the same as a regular API scrape response plus webhook information in the context part.
Webhooks are scoped per project per environment. Don't be angry if the API rejects when you pass webhook_name
, the webhook is not in the same project/environment as your API key.
If we can't notify the endpoint specified in your webhook settings.
If we failed to reach your application more than 100 times in a row, the system automatically disables it, and you will be notified. You can re-enable it from the UI at any moment.
Useful tools to develop locally :
For security reason, you can set define a secret, Scrapfly will reach you with an X-Scrapfly-Webhook-Secret
header containing
the defined secret
Following headers are added :
X-Scrapfly-Webhook-Env
: Related environment where webhook is triggeredX-Scrapfly-Webhook-Project
: Related project nameX-Scrapfly-Webhook-Secret
Related Secret to authenticate the origin of a callX-Scrapfly-Webhook-Name
Name of the webhookYou must create and configure webhook named example in order to play this example.
To notify the scrape response on your webhook, you simple need to pass the name of it on our API like in the example below.
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "webhook_name=sdfsdfsdf"
"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&webhook_name=sdfsdfsdf"
"api.scrapfly.io"
"/scrape"
key = ""
url = "https://httpbin.org/anything"
webhook_name = "sdfsdfsdf"
{
...
"context": {
...
"webhook": {
"name": "example",
"secret": "1a8c1001-5bc9-4b23-a830-31ac864cd18c",
"consecutive_failed_count": 0
}
...
}
...
}
All related errors are listed below. You can see full description and example of error response on Errors section
No additional fee applied on usage.