Debug

overview page of web interface
Debug tab in log inspection

Introduction

Debugging is a useful feature when you want investigate unexpected scrape content an replay it over and over or simply integrate the response in your test suite to handle it properly

Debug can be trigger automatically, on HTTP code =500, Anti Scraping Protection failure or manually by passing debug=true on API call

The complete response returned by our API is saved with the scrapped content. Like in the screen above, a link with the result is provided and accessible from outside

The debug content will be deleted when log retention policy expire

Without javascript rendering

Interactive Example: Debug Example
openapi openapi Debug Example Sign in 
            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=http%3A%2F%2Fhttpbin.org%2Fanything&debug=true
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=http%3A%2F%2Fhttpbin.org%2Fanything&debug=true

key
=
url
= http%3A%2F%2Fhttpbin.org%2Fanything
debug
= true

With javascript rendering

Interactive Example: Debug example with screenshot
openapi openapi Debug example with screenshot Sign in 
            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=http%3A%2F%2Fhttpbin.org%2Fanything&debug=true&render_js=true
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=http%3A%2F%2Fhttpbin.org%2Fanything&debug=true&render_js=true

key
=
url
= http%3A%2F%2Fhttpbin.org%2Fanything
debug
= true
render_js
= true

API Response

You need to pass your api key when requesting the url given in "response_url" like https://api.scrapfly.io/scrape/debug/ee8484c6-ee5f-4775-a665-0a2b57631c1c?key={{ YOUR_API_KEY }} to be authorized
openapi openapi API Response - Debug data | Json 
            ...
context {
    debug: {
        screenshot_url: "https://api.scrapfly.io/11cd6abe-5061-4dce-8d37-5d50e667a071/scrape/screenshot/ee8484c6-ee5f-4775-a665-0a2b57631c1c/debug",
        response_url: "https://api.scrapfly.io/scrape/debug/ee8484c6-ee5f-4775-a665-0a2b57631c1c"
    }
}
...

Integration