Debug

overview page of web interface
Debug tab in log inspection

Debugging is a valuable feature when you want to investigate unexpected scrape content and replay it over and over or simply integrate the response in your test suite to handle it properly.

The complete response returned by our API is saved with the scraped content. Like in the screen above, a link with the result is provided and accessible from outside.

If debug is activated beside javascript rendering, a screenshot is taken

The debug content will be deleted when the log retention policy expires

Usage

Sign in 
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=http://httpbin.dev/anything" \
--data-urlencode "proxy_pool=public_datacenter" \
--data-urlencode "debug=true"
"https://api.scrapfly.io/scrape?key=&url=http%3A%2F%2Fhttpbin.dev%2Fanything&proxy_pool=public_datacenter&debug=true"

"api.scrapfly.io"
"/scrape"

key         = "" 
url         = "http://httpbin.dev/anything" 
proxy_pool  = "public_datacenter" 
debug       = "true"

Example of API Response

You need to pass your API key when requesting the URL given in "response_url" like https://api.scrapfly.io/scrape/debug/ee8484c6-ee5f-4775-a665-0a2b57631c1c?key={{ YOUR_API_KEY }} to be authorized. 
...
context {
    debug: {
        // screenshot is available only if rendering javascript is activated
        screenshot_url: "https://api.scrapfly.io/11cd6abe-5061-4dce-8d37-5d50e667a071/scrape/screenshot/ee8484c6-ee5f-4775-a665-0a2b57631c1c/debug",
        response_url: "https://api.scrapfly.io/scrape/debug/ee8484c6-ee5f-4775-a665-0a2b57631c1c"
    }
}
...

Pricing

No additional fee.

Integration