# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Bring Your Own Proxy](https://scrapfly.io/docs/cloud-browser-api/bring-your-own-proxy) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Troubleshooting [ View as markdown ](https://scrapfly.io/?view=markdown) Copy for LLM Copy for LLM [ Open in ChatGPT ](https://chatgpt.com/?hints=search&prompt=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fscrape-api%2Ftroubleshoot%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Claude ](https://claude.ai/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fscrape-api%2Ftroubleshoot%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Perplexity ](https://www.perplexity.ai/search/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fscrape-api%2Ftroubleshoot%20so%20I%20can%20ask%20questions%20about%20it.) Troubleshooting Scrapfly scrapers is relatively easy as Scrapfly provides extensive logging. Let's take a look at some examples. To start the [Monitoring](https://scrapfly.io/docs/monitoring) dashboard displays all scrape requests and their details. Inspecting the scrape monitoring dashboard is the first step to troubleshooting and we'll cover this extensively in the following sections. For additional details make sure to enable the [debug](https://scrapfly.io/docs/scrape-api/getting-started#api_param_debug) parameter which will provide more details about the scrape requests and even capture screenshots when [render\_js](https://scrapfly.io/docs/scrape-api/getting-started#api_param_render_js) is enabled. ## Replicating with Player The first step to figuring out why a scrape request is not working as expected is to replicate the [Web Player](https://scrapfly.io/dashboard/playground/web-scraper) which allow easy and reliable scrape configuration 👉 If the [Web Player](https://scrapfly.io/dashboard/playground/web-scraper) works as expected, then the issue is likely related to the API call configuration. Ensure that the API call is configured as per [API Specification](https://scrapfly.io/docs/scrape-api/getting-started#spec). In particular, note that some API parameters need to be [url encoded](https://scrapfly.io/web-scraping-tools/urlencode). ## Timeout If you receive a lot of [`ERR::SCRAPE::OPERATION_TIMEOUT`](https://scrapfly.io/docs/scrape-api/error/ERR::SCRAPE::OPERATION_TIMEOUT) errors, inspect the following: - If `retry=false` is set, you can increase the `timeout` parameter - If `asp=true` and `retry=false` are set you can increase `timeout=60000` even `timeout=90000` - If the url you hit is redirected (like url ending with `/` or `wwww`) try to avoid unnecessary redirects and scrape the final urls to speed up scraping. - If you use `asp=true` you can replicate ASP fine-tuning as default scrape parameters. For example, ASP can upgrade requests to residential proxies and you can set this upgrade as default to save execution time. You can see ASP upgrades in the monitoring logs. You can the find [full documentation about timeout mechanism](https://scrapfly.io/docs/scrape-api/understand-timeout) ## Confirming Scrape Instructions Another way to troubleshoot this is to check the [Monitoring](https://scrapfly.io/docs/monitoring) dashboard and validate that Scrapfly received the same scrape instructions as you intended. 👈 If there's a mismatch, it's likely that the API call was incorrectly configured. If that's not the case then let's proceed further and take a look at potential causes. > Note that Scrapfly can alter parts of scrape details to bypass scraper blocking when [Anti Scraping Protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) is enabled. ## Unexpected Results The results you see in your browser are not always easy to replicate in a scraper. Here are the main causes that could explain **missing or different data**: #### Status Code Overview The first step is to ensure that the scrape request resulted in a **successful response**. To do this the `result.status_code`: ``` { [...] "result": { "status": "DONE", "status_code": 203, "success": true, "url": "https://httpbin.dev/status/203" [...] } ``` Here's a quick summary of the most common status codes and what are common causes for each: ##### 200 All 200 range codes (200, 201, 202 etc.) are considered success. If that's the case skip to the [Missing Scrape Details](#missing-req-data) section below. ##### 400 This error stands for client error and this likely means the scrape request is misconfigured: - Check the scrape URL - Check the scrape request headers - Check the scrape request body if it's a POST or PUT type request - See the [Missing Scrape Details](#missing-req-data) section below ##### 404 The page is **not found** and this likely means the scrape URL is invalid. Typo in the URL is most likely cause though alternatively, this can also mean request misconfiguration. See the [Missing Scrape Details](#missing-req-data) section below. ##### 410 The page is **gone** and this likely means the scrape URL has become invalid. This is common when scraping pages with an expiration date like second hand listings or advertisements. ##### 405 The page doesn't accept current HTTP method. This can be caused by sending POST-type request to GET endpoints and vice versa. ##### 406 The page doesn't accept current content type. This can be caused by sending JSON-type request to HTML endpoints and vice versa or setting invalid `Accept` header. #### Missing Scrape Details The most common reason for missing data is that the scraper is missing configuration details that are required to fully load the page. When replicating requests from web browser in Scrapfly it's important to match all request details. ##### Related [Understanding Websites Academy](https://scrapfly.io/academy/reverse-engineering) [How to send cookies or custom request headers?](https://scrapfly.io/docs/scrape-api/faq#headers-and-cookies) [What can cause missing URL parameters?](https://scrapfly.io/docs/scrape-api/faq#missing-parameter) ##### URL Parameters URL parameters are everything after the `?` symbol and optimally this should match what we see in the browser. For example in the URL `https://web-scraping.dev/product/2?variant=one&COLOR=dark%20blue` we should keep both `variant=one` and `COLOR=dark%20blue` parameters in the scrape request as they appear in the URL including: - Parameter order. Here, "variant" then "COLOR" - Parameter name spelling and case. Here, "COLOR" is uppercase - Parameter value encoding and formatting (if any). Here, the color value is url encoded as "dark%20blue" Some non-functional parameters like analytics tracking parameters should be ignored. These parameters often appear as non-sensical IDs (like `?tid=cfa44df`) and can be easily confirmed if the website functionality remains the same when they are removed. ##### Request Headers While Scrapfly configures all headers related to fingerprinting and blocking it cannot predict all custom headers for all websites. This is particularly important when websites use customer headers that are usually identified with `X-` prefix - these should be replicated and included in scrape requests. ##### Related [Scrapeground CSRF Exercise](https://scrapfly.io/scrapeground/headers/csrf) ##### Cookies Some pages can be cookie-locked and require cookies set from previous requests. The easiest way to handle this is to use Scrapfly [Session](https://scrapfly.io/docs/scrape-api/session) and request the pages in the required order. ##### Related [Scrapeground Cookies Exercise](https://scrapfly.io/scrapeground/cookies) #### Dynamic Websites The website could be **dynamically loaded** through browser javascript. If you're scraping without [render\_js](https://scrapfly.io/docs/scrape-api/getting-started#api_param_render_js) parameter Scrapfly is not executing javascript which can cause the said data difference. If you are using `render_js` then ensure the scraper is waiting for the website to fully load using the [wait\_for\_selector](https://scrapfly.io/docs/scrape-api/getting-started#api_param_wait_for_selector) or [rendering\_wait](https://scrapfly.io/docs/scrape-api/getting-started#api_param_rendering_wait) parameters. ##### Related [Dynamic Scraping Academy](https://scrapfly.io/academy/dynamic-scraping) [How to scrape with headless browsers?](https://scrapfly.io/docs/scrape-api/faq#scrape-with-headless-browsers) #### Geo Location Data could be different or missing because of a different **proxy country**. By default, Scrapfly selects a fitting proxy randomly which might not always match the desired scrape region. Try changing the proxy [country](https://scrapfly.io/docs/scrape-api/getting-started#api_param_country) parameter to match your region, e.g. `country=us`. ##### Related [Scraper Proxies Academy](https://scrapfly.io/academy/proxies) [What proxy countries does Scrapfly support?](https://scrapfly.io/docs/scrape-api/faq#proxy-countries) [How to set proxy country?](https://scrapfly.io/docs/scrape-api/faq#proxy-set-country) #### Scraper Blocking Another cause could be **anti-bot** measures which are designed to block scrapers. For this, make sure [Anti Scraping Protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) is enabled. Additionally, you can improve success chances by: - Enabling browser usage by default `render_js=true` [ ](https://scrapfly.io/docs/scrape-api/javascript-rendering) - Enabling residential proxy network `proxy_pool=public_residential_pool` [ ](https://scrapfly.io/docs/scrape-api/proxy#api) - Ensuring the scrape config matches web browser requests as closely as possible - request headers, POST content etc. For more see the related FAQ entries. 👉 ##### Related [Scraper Blocking Academy](https://scrapfly.io/academy/scraper-blocking) [Why does ASP fail to bypass protection?](https://scrapfly.io/docs/scrape-api/faq#asp-failing) ## Errors Scrapfly uses a comprehensive error code status system that indicates exactly what went wrong with a scrape request and error details can be accessed through Scrapfly response `result.error` field: ``` { "config": { ... }, "context": { ... }, "result": { [...], "status": "DONE", "success": false, "reason": null, "error": { "http_code": 429, "code": "ERR::THROTTLE::MAX_REQUEST_RATE_EXCEEDED", "description": "Your scrape request as been throttle. Too much request during the last minute. If it's not expected, please check your throttle configuration for the given project and env", "error_id": "9993a546-b899-4927-b788-04f5c4e473d5", "message": "Max request rate exceeded", "scrape_id": "7c61352c-f1a7-4ea6-a0b8-198d7ac6fe1a", "retryable": false, "doc_url": "https://scrapfly.io/docs/scrape-api/error/ERR::THROTTLE::MAX_REQUEST_RATE_EXCEEDED" }, [...], } } ``` This field contains all of the information needed to troubleshoot the error. In particular, error `code` and `http_code` can be used to troubleshoot any error page. The http and error codes are defined in the **[Errors](https://scrapfly.io/docs/scrape-api/errors)** documentation or can be looked up through using the quick search command. For example, http status code: 422 will look up all errors related to http status code `422` and ERR::SCRAPE::UPSTREAM\_WEBSITE\_ERROR will look up all pages related to this error. ## Scraping Costs To troubleshoot billing issues, first check the `cost` field in the [Monitoring](https://scrapfly.io/docs/monitoring) dashboard. This field breaks down all the credit costs used for this scrape request. 👈 Note that scrape cost can vary for each scrape request depending on the scrape details. Many costs like [Anti Scraping Protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) are only billed when use is required and tech is reused when possible with reduced charge. ##### Related FAQ [How to estimate credit cost for a scrape?](https://scrapfly.io/docs/scrape-api/faq#billing-calculation) [How to control Scrapfly credit spending?](https://scrapfly.io/docs/scrape-api/faq#spending) [What happens if I run out of Scrapfly credits?](https://scrapfly.io/docs/scrape-api/faq#extra-use-kickin) [More FAQ](https://scrapfly.io/docs/scrape-api/faq) [Contact Support](https://scrapfly.io/docs/support)