API Errors
Introduction
If you want to port those definitions into your application, you can checkout the Exportable Definition section to retrieve the JSON describing errors.
Only successful requests are counted into your quota. Each time a bad status code from an upstream website or error is raised, you are not charged for it.
Generic API Error
Example of API error response :
{
status: "error",
http_code: 401,
reason: "Unauthorized",
error_id: "301e2d9e-b4f5-4289-85ea-e452143338df",
message: "Invalid API key"
}
HTTP 400
Bad Request: Parameters sent to the API are incorrect. Check out the related documentation to figure out the error.
HTTP 404
Not Found: API URL might have a typo or be incorrect. Check out the related documentation to figure out the error.
HTTP 422
Unprocessable Entity: Your request was correct, but for some reason, we cannot handle it. Most of the time, the entity which you want to update/delete has already been processed.
HTTP 429
Too Many Request: API endpoint, which should be called with a high frequency, are throttled internally to prevent service disruption. If this happened too many times, your account would be suspended.
HTTP 502
Web service that exposes Scrapfly to the internet is in trouble, and we have been alerted of the issue.
HTTP 503
Service Temporary Unavailable: Scrapfly might run in degraded mode, or maintenance was scheduled to upgrade our service.
Web Scraping API Errors
Error is present in response body response['result']['error']
.
{
"config": { ... },
"context": { ... },
"result": {
[...],
"status": "DONE",
"success": false,
"reason": null,
"error": {
"http_code": 429,
"code": "ERR::THROTTLE::MAX_REQUEST_RATE_EXCEEDED",
"description": "Your scrape request as been throttle. Too much request during the last minute. If it's not expected, please check your throttle configuration for the given project and env",
"error_id": "9993a546-b899-4927-b788-04f5c4e473d5",
"message": "Max request rate exceeded",
"scrape_id": "7c61352c-f1a7-4ea6-a0b8-198d7ac6fe1a",
"retryable": false,
"doc_url": "https://scrapfly.io/docs/scrape-api/error/ERR::THROTTLE::MAX_REQUEST_RATE_EXCEEDED"
},
[...],
}
}
Error is also present in response headers
x-scrapfly-reject-code: ERR::SCRAPE::DNS_NAME_NOT_RESOLVED
x-scrapfly-reject-description: The DNS of upstream website is not resolving or not responding
x-scrapfly-reject-doc: https://scrapfly.io/docs/scrape-api/error/ERR::SCRAPE::DNS_NAME_NOT_RESOLVED
x-scrapfly-reject-http-code: 523
x-scrapfly-reject-id: 5556636c-ac89-417f-b645-02c32905e39a
x-scrapfly-reject-retryable: no
ERR::SCRAPE::SSL_ERROR
Error due to SSL, mostly due to handshake error
- Retryable: No
- HTTP status code: 500
ERR::SCRAPE::DOMAIN_NOT_ALLOWED
The Domain targeted is not allowed or restricted
- Retryable: No
- HTTP status code: 403
ERR::SCRAPE::SCENARIO_DEADLINE_OVERFLOW
Submitted scenario would require more than 30s to complete
- Retryable: No
- HTTP status code: 400
ERR::SCRAPE::UPSTREAM_TOO_MANY_REDIRECT
The upstream website makes too many redirects
- Retryable: No
- HTTP status code: 500
ERR::SCRAPE::DOM_SELECTOR_NOT_FOUND
The requested DOM selected was not found in rendered content within 15s
- Retryable: No
- HTTP status code: 400
ERR::SCRAPE::DOM_SELECTOR_INVISIBLE
The requested DOM selected is invisible (Mostly issued when element is targeted for screenshot)
- Retryable: No
- HTTP status code: 400
ERR::SCRAPE::QUOTA_LIMIT_REACHED
You reach your scrape quota plan for the month. You can upgrade your plan if you want increase the quota
- Retryable: No
- HTTP status code: 429
ERR::SCRAPE::OPERATION_TIMEOUT
This is a generic error for when timeout occur. It happened when internal operation took too much time
- Retryable: Yes
- HTTP status code: 408
ERR::SCRAPE::PROJECT_QUOTA_LIMIT_REACHED
The limit set to the current project has been reached
- Retryable: No
- HTTP status code: 429
ERR::SCRAPE::TOO_MANY_CONCURRENT_REQUEST
You reach concurrent limit of scrape request of your current plan
- Retryable: Yes
- HTTP status code: 429
ERR::SCRAPE::NO_BROWSER_AVAILABLE
No browser available in the pool
- Retryable: Yes
- HTTP status code: 503
ERR::SCRAPE::UNABLE_TO_TAKE_SCREENSHOT
Unable to take screenshot, happened when renderer encounter bad formatted code
- Retryable: Yes
- HTTP status code: 500
ERR::SCRAPE::UPSTREAM_TIMEOUT
Upstream website made too much time to response
- Retryable: No
- HTTP status code: 408
ERR::SCRAPE::JAVASCRIPT_EXECUTION
The javascript to execute goes wrong, please read the associated message to figure out the problem
- Retryable: No
- HTTP status code: 500
ERR::SCRAPE::BAD_UPSTREAM_RESPONSE
Upstream respond with http code >= 299
- Retryable: No
- HTTP status code: 200
ERR::SCRAPE::DRIVER_CRASHED
Driver used to perform the scrape can crash for many reason
- Retryable: Yes
- HTTP status code: 503
ERR::SCRAPE::DRIVER_TIMEOUT
Driver timeout - No response received
- Retryable: Yes
- HTTP status code: 408
Proxy
ERR::PROXY::POOL_NOT_AVAILABLE_FOR_TARGET
Proxy Pool is not available for target - most of the time datacenter network is not available for the target
- Retryable: No
- HTTP status code: 503
ERR::PROXY::RESOURCES_SATURATION
Proxy are saturated for the desired country, you can try on other countries. They will come back as soon as possible
- Retryable: Yes
- HTTP status code: 503
ERR::PROXY::NOT_REACHABLE
Proxy was not reachable, it can happened when network issue or proxy itself is in trouble
- Retryable: Yes
- HTTP status code: 503
ERR::PROXY::UNAVAILABLE
Proxy network is unavailable - or too many attempt has been done on pool
- Retryable: Yes
- HTTP status code: 503
ERR::PROXY::POOL_NOT_FOUND
Provided Proxy Pool Name do not exists
- Retryable: No
- HTTP status code: 400
ERR::PROXY::POOL_UNAVAILABLE_COUNTRY
Country not available for given proxy pool
- Retryable: No
- HTTP status code: 400
Throttle
ERR::THROTTLE::MAX_CONCURRENT_REQUEST_EXCEEDED
Your scrape request has been throttled. Too many concurrent access to the upstream. If it's not expected, please check your throttle configuration for the given project and env.
- Retryable: Yes
- HTTP status code: 429
ERR::THROTTLE::MAX_REQUEST_RATE_EXCEEDED
Your scrape request as been throttle. Too much request during the last minute. If it's not expected, please check your throttle configuration for the given project and env
- Retryable: Yes
- HTTP status code: 429
Anti Scraping Protection (ASP)
ERR::ASP::PROTECTION_FAILED
Something wrong happened with specific shield of anti bot protection. We will figure out to fix the problem as soon as possible. Sometimes Anti bot protection change their technics and we need to adapt our system
- Retryable: No
- HTTP status code: 500
ERR::ASP::CAPTCHA_ERROR
Something wrong happened with the captcha. We will figure out to fix the problem as soon as possible
- Retryable: No
- HTTP status code: 500
ERR::ASP::UNABLE_TO_SOLVE_CAPTCHA
Despite our effort, we were unable to solve the captcha. It can happened sporadically, please retry
- Retryable: Yes
- HTTP status code: 500
ERR::ASP::CAPTCHA_TIMEOUT
The budgeted time to solve the captcha is reached
- Retryable: Yes
- HTTP status code: 408
ERR::ASP::UPSTREAM_UNEXPECTED_RESPONSE
The response given by the upstream after challenge resolution is not expected. Our team has been alerted
- Retryable: No
- HTTP status code: 500
ERR::ASP::SHIELD_ERROR
The ASP encounter an unexpected problem. We will fix it as soon as possible. Our team has been alerted
- Retryable: No
- HTTP status code: 500
ERR::ASP::SHIELD_FAILED
The ASP shield failed to solve the challenge against the anti scrapping protection
- Retryable: Yes
- HTTP status code: 500
ERR::ASP::TIMEOUT
The ASP made too much time to solve or respond
- Retryable: Yes
- HTTP status code: 408
Schedule
Webhook
ERR::WEBHOOK::NOT_FOUND
Unable to find the given webhook for the current project / env
- Retryable: No
- HTTP status code: 400
ERR::WEBHOOK::DISABLED
Given webhook is disabled, please check out your webhook configuration for the current project / env
- Retryable: No
- HTTP status code: 500
ERR::WEBHOOK::MAX_RETRY
Maximum retry exceeded on your webhook
- Retryable: No
- HTTP status code: 429
ERR::WEBHOOK::QUEUE_FULL
You reach the limit of 1000/1000 scheduled webhook
- Retryable: Yes
- HTTP status code: 429
Session
ERR::SESSION::CONCURRENT_ACCESS
Concurrent access to the session has been tried. If your spider run on distributed architecture, check if the correlation id is correctly configured
- Retryable: Yes
- HTTP status code: 429
Exportable Definition
If you want to handle errors from your application without copy-pasting the whole error definition into your application to match errors, here is a portable JSON of error definition:
{
"ERR::PROXY::POOL_NOT_AVAILABLE_FOR_TARGET": {
"description": "Proxy Pool is not available for target - most of the time datacenter network is not available for the target",
"retryable": false,
"response_code": 503,
"docs": [],
"code": "ERR::PROXY::POOL_NOT_AVAILABLE_FOR_TARGET"
},
"ERR::PROXY::RESOURCES_SATURATION": {
"description": "Proxy are saturated for the desired country, you can try on other countries. They will come back as soon as possible",
"retryable": true,
"response_code": 503,
"docs": [],
"code": "ERR::PROXY::RESOURCES_SATURATION"
},
"ERR::PROXY::NOT_REACHABLE": {
"description": "Proxy was not reachable, it can happened when network issue or proxy itself is in trouble",
"retryable": true,
"response_code": 503,
"docs": [],
"code": "ERR::PROXY::NOT_REACHABLE"
},
"ERR::PROXY::UNAVAILABLE": {
"description": "Proxy network is unavailable - or too many attempt has been done on pool",
"retryable": true,
"response_code": 503,
"docs": [],
"code": "ERR::PROXY::UNAVAILABLE"
},
"ERR::PROXY::POOL_NOT_FOUND": {
"description": "Provided Proxy Pool Name do not exists",
"retryable": false,
"response_code": 400,
"docs": {
"Checkout Proxy Documentation": "https://scrapfly.io/docs/scrape-api/proxy"
},
"code": "ERR::PROXY::POOL_NOT_FOUND"
},
"ERR::PROXY::POOL_UNAVAILABLE_COUNTRY": {
"description": "Country not available for given proxy pool",
"retryable": false,
"response_code": 400,
"docs": {
"Checkout Proxy Documentation": "https://scrapfly.io/docs/scrape-api/proxy"
},
"code": "ERR::PROXY::POOL_UNAVAILABLE_COUNTRY"
},
"ERR::THROTTLE::MAX_CONCURRENT_REQUEST_EXCEEDED": {
"description": "Your scrape request has been throttled. Too many concurrent access to the upstream. If it's not expected, please check your throttle configuration for the given project and env.",
"retryable": true,
"response_code": 429,
"docs": [],
"code": "ERR::THROTTLE::MAX_CONCURRENT_REQUEST_EXCEEDED"
},
"ERR::THROTTLE::MAX_REQUEST_RATE_EXCEEDED": {
"description": "Your scrape request as been throttle. Too much request during the last minute. If it's not expected, please check your throttle configuration for the given project and env",
"retryable": true,
"response_code": 429,
"docs": [],
"code": "ERR::THROTTLE::MAX_REQUEST_RATE_EXCEEDED"
},
"ERR::ASP::PROTECTION_FAILED": {
"description": "Something wrong happened with specific shield of anti bot protection. We will figure out to fix the problem as soon as possible. Sometimes Anti bot protection change their technics and we need to adapt our system",
"retryable": false,
"response_code": 500,
"docs": [],
"code": "ERR::ASP::PROTECTION_FAILED"
},
"ERR::ASP::CAPTCHA_ERROR": {
"description": "Something wrong happened with the captcha. We will figure out to fix the problem as soon as possible",
"retryable": false,
"response_code": 500,
"docs": [],
"code": "ERR::ASP::CAPTCHA_ERROR"
},
"ERR::ASP::UNABLE_TO_SOLVE_CAPTCHA": {
"description": "Despite our effort, we were unable to solve the captcha. It can happened sporadically, please retry",
"retryable": true,
"response_code": 500,
"docs": [],
"code": "ERR::ASP::UNABLE_TO_SOLVE_CAPTCHA"
},
"ERR::ASP::CAPTCHA_TIMEOUT": {
"description": "The budgeted time to solve the captcha is reached",
"retryable": true,
"response_code": 408,
"docs": [],
"code": "ERR::ASP::CAPTCHA_TIMEOUT"
},
"ERR::ASP::UPSTREAM_UNEXPECTED_RESPONSE": {
"description": "The response given by the upstream after challenge resolution is not expected. Our team has been alerted",
"retryable": false,
"response_code": 500,
"docs": [],
"code": "ERR::ASP::UPSTREAM_UNEXPECTED_RESPONSE"
},
"ERR::ASP::SHIELD_ERROR": {
"description": "The ASP encounter an unexpected problem. We will fix it as soon as possible. Our team has been alerted",
"retryable": false,
"response_code": 500,
"docs": [],
"code": "ERR::ASP::SHIELD_ERROR"
},
"ERR::ASP::SHIELD_EXPIRED": {
"description": "The ASP shield previously set is expired, you must retry.",
"context": "This issue should not be raised on your side, it's managed internally.",
"retryable": true,
"response_code": 419,
"docs": [],
"code": "ERR::ASP::SHIELD_EXPIRED"
},
"ERR::ASP::SHIELD_FAILED": {
"description": "The ASP shield failed to solve the challenge against the anti scrapping protection",
"retryable": true,
"response_code": 500,
"docs": [],
"code": "ERR::ASP::SHIELD_FAILED"
},
"ERR::ASP::TIMEOUT": {
"description": "The ASP made too much time to solve or respond",
"retryable": true,
"response_code": 408,
"docs": [],
"code": "ERR::ASP::TIMEOUT"
},
"ERR::SCHEDULE::DISABLED": {
"description": "The targeted schedule has been disabled",
"retryable": false,
"response_code": 504,
"docs": [],
"code": "ERR::SCHEDULE::DISABLED"
},
"ERR::SCRAPE::SSL_ERROR": {
"description": "Error due to SSL, mostly due to handshake error",
"retryable": false,
"response_code": 500,
"docs": [],
"code": "ERR::SCRAPE::SSL_ERROR"
},
"ERR::SCRAPE::SCENARIO_EXECUTION": {
"description": "Javascript Scenario Failed",
"retryable": false,
"response_code": 400,
"docs": [],
"code": "ERR::SCRAPE::SCENARIO_EXECUTION"
},
"ERR::SCRAPE::SCENARIO_TIMEOUT": {
"description": "Javascript Scenario Timeout",
"retryable": true,
"response_code": 408,
"docs": [],
"code": "ERR::SCRAPE::SCENARIO_TIMEOUT"
},
"ERR::SCRAPE::DOMAIN_NOT_ALLOWED": {
"description": "The Domain targeted is not allowed or restricted",
"retryable": false,
"response_code": 403,
"docs": [],
"code": "ERR::SCRAPE::DOMAIN_NOT_ALLOWED"
},
"ERR::SCRAPE::SCENARIO_DEADLINE_OVERFLOW": {
"description": "Submitted scenario would require more than 30s to complete",
"retryable": false,
"response_code": 400,
"docs": [],
"code": "ERR::SCRAPE::SCENARIO_DEADLINE_OVERFLOW"
},
"ERR::SCRAPE::UPSTREAM_TOO_MANY_REDIRECT": {
"description": "The upstream website makes too many redirects",
"retryable": false,
"response_code": 500,
"docs": [],
"code": "ERR::SCRAPE::UPSTREAM_TOO_MANY_REDIRECT"
},
"ERR::SCRAPE::DOM_SELECTOR_NOT_FOUND": {
"description": "The requested DOM selected was not found in rendered content within 15s",
"retryable": false,
"response_code": 400,
"docs": [],
"code": "ERR::SCRAPE::DOM_SELECTOR_NOT_FOUND"
},
"ERR::SCRAPE::DOM_SELECTOR_INVISIBLE": {
"description": "The requested DOM selected is invisible (Mostly issued when element is targeted for screenshot)",
"retryable": false,
"response_code": 400,
"docs": [],
"code": "ERR::SCRAPE::DOM_SELECTOR_INVISIBLE"
},
"ERR::SCRAPE::QUOTA_LIMIT_REACHED": {
"description": "You reach your scrape quota plan for the month. You can upgrade your plan if you want increase the quota",
"retryable": false,
"response_code": 429,
"docs": [],
"code": "ERR::SCRAPE::QUOTA_LIMIT_REACHED"
},
"ERR::SCRAPE::OPERATION_TIMEOUT": {
"description": "This is a generic error for when timeout occur. It happened when internal operation took too much time",
"retryable": true,
"response_code": 408,
"docs": [],
"code": "ERR::SCRAPE::OPERATION_TIMEOUT"
},
"ERR::SCRAPE::PROJECT_QUOTA_LIMIT_REACHED": {
"description": "The limit set to the current project has been reached",
"retryable": false,
"response_code": 429,
"docs": [],
"code": "ERR::SCRAPE::PROJECT_QUOTA_LIMIT_REACHED"
},
"ERR::SCRAPE::TOO_MANY_CONCURRENT_REQUEST": {
"description": "You reach concurrent limit of scrape request of your current plan",
"retryable": true,
"response_code": 429,
"docs": [],
"code": "ERR::SCRAPE::TOO_MANY_CONCURRENT_REQUEST"
},
"ERR::SCRAPE::NO_BROWSER_AVAILABLE": {
"description": "No browser available in the pool",
"retryable": true,
"response_code": 503,
"docs": [],
"code": "ERR::SCRAPE::NO_BROWSER_AVAILABLE"
},
"ERR::SCRAPE::UNABLE_TO_TAKE_SCREENSHOT": {
"description": "Unable to take screenshot, happened when renderer encounter bad formatted code",
"retryable": true,
"response_code": 500,
"docs": [],
"code": "ERR::SCRAPE::UNABLE_TO_TAKE_SCREENSHOT"
},
"ERR::SCRAPE::UPSTREAM_TIMEOUT": {
"description": "Upstream website made too much time to response",
"retryable": false,
"response_code": 408,
"docs": [],
"code": "ERR::SCRAPE::UPSTREAM_TIMEOUT"
},
"ERR::SCRAPE::JAVASCRIPT_EXECUTION": {
"description": "The javascript to execute goes wrong, please read the associated message to figure out the problem",
"retryable": false,
"response_code": 500,
"docs": {
"Checkout Javascript Rendering Documentation": "https://scrapfly.io/docs/scrape-api/javascript-rendering"
},
"code": "ERR::SCRAPE::JAVASCRIPT_EXECUTION"
},
"ERR::SCRAPE::BAD_UPSTREAM_RESPONSE": {
"description": "Upstream respond with http code >= 299",
"retryable": false,
"context": "HTTP Status code relate the state of the service. Forwarding you the upstream response code as our api response code is a non sense, it will create confusion & complexity between regular Scrapfly error and do not represent the correct current state of our service. For simplicity of client-side implementation, we decided to treat bad upstream response code as regular error with same structure and answer with a 200. Not conventional but we think it's the best solution",
"response_code": 200,
"docs": [],
"code": "ERR::SCRAPE::BAD_UPSTREAM_RESPONSE"
},
"ERR::SCRAPE::DNS_NAME_NOT_RESOLVED": {
"description": "The DNS of upstream website is not resolving or not responding",
"retryable": false,
"response_code": 523,
"docs": [],
"code": "ERR::SCRAPE::DNS_NAME_NOT_RESOLVED"
},
"ERR::SCRAPE::BAD_PROTOCOL": {
"description": "The protocol is not supported",
"retryable": false,
"response_code": 500,
"docs": [],
"code": "ERR::SCRAPE::BAD_PROTOCOL"
},
"ERR::SCRAPE::NETWORK_SERVER_DISCONNECTED": {
"description": "Server of upstream website closed unexpectedly the connection",
"retryable": false,
"response_code": 499,
"docs": [],
"code": "ERR::SCRAPE::NETWORK_SERVER_DISCONNECTED"
},
"ERR::SCRAPE::NETWORK_ERROR": {
"description": "Network error happened between Scrapfly server and remote server",
"context": "This king of error are automatically retried internally if auto retry is not disabled. If you see this error, it means it already has been tried 5 times",
"retryable": true,
"response_code": 499,
"docs": [],
"code": "ERR::SCRAPE::NETWORK_ERROR"
},
"ERR::SCRAPE::DRIVER_CRASHED": {
"description": "Driver used to perform the scrape can crash for many reason",
"retryable": true,
"response_code": 503,
"docs": [],
"code": "ERR::SCRAPE::DRIVER_CRASHED"
},
"ERR::SCRAPE::DRIVER_TIMEOUT": {
"description": "Driver timeout - No response received",
"retryable": true,
"response_code": 408,
"docs": [],
"code": "ERR::SCRAPE::DRIVER_TIMEOUT"
},
"ERR::WEBHOOK::NOT_FOUND": {
"description": "Unable to find the given webhook for the current project / env",
"retryable": false,
"response_code": 400,
"docs": {
"Checkout Webhook Documentation": "https://scrapfly.io/docs/scrape-api/webhook"
},
"code": "ERR::WEBHOOK::NOT_FOUND"
},
"ERR::WEBHOOK::DISABLED": {
"description": "Given webhook is disabled, please check out your webhook configuration for the current project / env",
"retryable": false,
"response_code": 500,
"docs": {
"Checkout Webhook Documentation": "https://scrapfly.io/docs/scrape-api/webhook"
},
"code": "ERR::WEBHOOK::DISABLED"
},
"ERR::WEBHOOK::MAX_RETRY": {
"description": "Maximum retry exceeded on your webhook",
"retryable": false,
"response_code": 429,
"docs": {
"Checkout Webhook Documentation": "https://scrapfly.io/docs/scrape-api/webhook"
},
"code": "ERR::WEBHOOK::MAX_RETRY"
},
"ERR::WEBHOOK::QUEUE_FULL": {
"description": "You reach the limit of 1000/1000 scheduled webhook",
"retryable": true,
"response_code": 429,
"docs": {
"Checkout Webhook Documentation": "https://scrapfly.io/docs/scrape-api/webhook"
},
"code": "ERR::WEBHOOK::QUEUE_FULL"
},
"ERR::SESSION::CONCURRENT_ACCESS": {
"description": "Concurrent access to the session has been tried. If your spider run on distributed architecture, check if the correlation id is correctly configured",
"retryable": true,
"response_code": 429,
"docs": {
"Checkout Session Documentation": "https://scrapfly.io/docs/scrape-api/session"
},
"code": "ERR::SESSION::CONCURRENT_ACCESS"
}
}