Debug
Scrapfly's debug feature allows debugging each API call by:
- Storing the scraped result for 1+ weeks (based on log retention policy)
- Replaying any debug API call through web dashboard
- Screenshot capturing for javascript rendering enabled API calls
We recommend using debug
feature during scraper development process, as it makes it easier to investigate any scraper issues.
The stored debug content will be deleted when the log retention policy expires
The debug details can be found in each monitoring log page Debug
tab:
Usage
Debug has to be enabled per request basis using the debug parameter.
If used together with the render_js=true
parameter the debugger will also attempt to capture screenshots:
import requests
url = "https://api.scrapfly.io/scrape?debug=true&render_js=true&key=__API_KEY__&url=https%3A%2F%2Fweb-scraping.dev%2Fproduct%2F1"
response = requests.request("GET", url)
data = response.json()
print(data)
print(data['result'])
"https://api.scrapfly.io/scrape?debug=true&render_js=true&key=&url=https%3A%2F%2Fweb-scraping.dev%2Fproduct%2F1"
"api.scrapfly.io"
"/scrape"
debug = "true"
render_js = "true"
key = ""
url = "https://web-scraping.dev/product/1"
Example of API Response
To access the debug results via provided urls the?key
parameter should be added with your API key. For example:https://api.scrapfly.io/scrape/debug/ee8484c6-ee5f-4775-a665-0a2b57631c1c?key={{ YOUR_API_KEY }}
Pricing
No additional fee.