Debug

overview page of web interface
Debug tab in log inspection

Introduction

Debugging is a valuable feature when you want to investigate unexpected scrape content and replay it over and over or simply integrate the response in your test suite to handle it properly.

Debug can be trigger automatically, on HTTP code =500, Anti Scraping Protection failure or manually by passing debug=true on API call.

The complete response returned by our API is saved with the scrapped content. Like in the screen above, a link with the result is provided and accessible from outside.

The debug content will be deleted when the log retention policy expires

Without javascript rendering

Interactive Example: Debug Example
openapi openapi Debug Example Sign in 
            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=http%3A%2F%2Fhttpbin.org%2Fanything&debug=true
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=http%3A%2F%2Fhttpbin.org%2Fanything&debug=true

key
=
url
= http%3A%2F%2Fhttpbin.org%2Fanything
debug
= true

With javascript rendering

Interactive Example: Debug example with screenshot
openapi openapi Debug example with screenshot Sign in 
            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=http%3A%2F%2Fhttpbin.org%2Fanything&debug=true&render_js=true
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=http%3A%2F%2Fhttpbin.org%2Fanything&debug=true&render_js=true

key
=
url
= http%3A%2F%2Fhttpbin.org%2Fanything
debug
= true
render_js
= true

API Response

You need to pass your API key when requesting the URL given in "response_url" like https://api.scrapfly.io/scrape/debug/ee8484c6-ee5f-4775-a665-0a2b57631c1c?key={{ YOUR_API_KEY }} to be authorized.
openapi openapi API Response - Debug data | Json 
            ...
context {
    debug: {
        screenshot_url: "https://api.scrapfly.io/11cd6abe-5061-4dce-8d37-5d50e667a071/scrape/screenshot/ee8484c6-ee5f-4775-a665-0a2b57631c1c/debug",
        response_url: "https://api.scrapfly.io/scrape/debug/ee8484c6-ee5f-4775-a665-0a2b57631c1c"
    }
}
...

Integration