API Errors

Introduction

If you want to port those definitions into your application, you can checkout the Exportable Definition section to retrieve the JSON describing errors.

Generic API Error

Example of API error response :

{
    status: "error",
    http_code: 401,
    reason: "Unauthorized",
    error_id: "301e2d9e-b4f5-4289-85ea-e452143338df",
    message: "Invalid API key"
}

HTTP 400

Bad Request: Parameters sent to the API are incorrect. Check out the related documentation to figure out the error.

HTTP 404

Not Found: API URL might have a typo or be incorrect. Check out the related documentation to figure out the error.

HTTP 422

Unprocessable Entity: Your request was correct, but for some reason, we cannot handle it. Most of the time, the entity which you want to update/delete has already been processed.

HTTP 429

Too Many Request: API endpoint, which should be called with a high frequency, are throttled internally to prevent service disruption. If this happened too many times, your account would be suspended.

HTTP 500

Internal Server Error: Scrapfly is in trouble, and we have been alerted of the issue.

HTTP 502

Web service that exposes Scrapfly to the internet is in trouble, and we have been alerted of the issue.

HTTP 503

Service Temporary Unavailable: Scrapfly might run in degraded mode, or maintenance was scheduled to upgrade our service.

HTTP 504

Scrapfly is not reachable or takes too much time to respond.

Web Scraping API Errors

Scrapfly provides API errors in two locations - in the response body and the response headers:

The error details are present in the response body result.error key.

The error details are also present in the response x-scrapfly-reject-* headers.

Error Definitions

ERR::SCRAPE::BAD_PROTOCOL

The protocol is not supported only http:// or https:// are supported

ERR::SCRAPE::BAD_UPSTREAM_RESPONSE

The website you target respond with an unexpected status code (>400)

ERR::SCRAPE::COST_BUDGET_LIMIT

Cost budget has been reached, you must increase the budget to pass this target

ERR::SCRAPE::DNS_NAME_NOT_RESOLVED

The DNS of the targeted website is not resolving or not responding

ERR::SCRAPE::DOMAIN_NOT_ALLOWED

The Domain targeted is not allowed or restricted

ERR::SCRAPE::DOM_SELECTOR_INVALID

The DOM Selector is invalid

ERR::SCRAPE::DOM_SELECTOR_INVISIBLE

The requested DOM selected is invisible (Mostly issued when element is targeted for screenshot)

ERR::SCRAPE::DOM_SELECTOR_NOT_FOUND

The requested DOM selected was not found in rendered content within 15s

ERR::SCRAPE::DRIVER_CRASHED

Driver used to perform the scrape can crash for many reason

ERR::SCRAPE::DRIVER_INSUFFICIENT_RESOURCES

Driver do not have enough resource to render the page correctly

ERR::SCRAPE::DRIVER_TIMEOUT

Driver timeout - No response received

ERR::SCRAPE::FORMAT_CONVERSION_ERROR

Response format conversion failed, unsupported input content type

ERR::SCRAPE::JAVASCRIPT_EXECUTION

The javascript to execute goes wrong, please read the associated message to figure out the problem

ERR::SCRAPE::NETWORK_ERROR

Network error happened between Scrapfly server and remote server

ERR::SCRAPE::NETWORK_SERVER_DISCONNECTED

Server of upstream website closed unexpectedly the connection

ERR::SCRAPE::NO_BROWSER_AVAILABLE

No browser available in the pool

ERR::SCRAPE::OPERATION_TIMEOUT

This is a generic error for when timeout occur. It happened when internal operation took too much time

ERR::SCRAPE::PROJECT_QUOTA_LIMIT_REACHED

The limit set to the current project has been reached

ERR::SCRAPE::QUOTA_LIMIT_REACHED

You reach your scrape quota plan for the month. You can upgrade your plan if you want increase the quota

ERR::SCRAPE::SCENARIO_DEADLINE_OVERFLOW

Submitted scenario would require more than 30s to complete

ERR::SCRAPE::SCENARIO_EXECUTION

Javascript Scenario Failed

ERR::SCRAPE::SCENARIO_TIMEOUT

Javascript Scenario Timeout

ERR::SCRAPE::SSL_ERROR

Upstream website have SSL error

ERR::SCRAPE::TOO_MANY_CONCURRENT_REQUEST

You reach concurrent limit of scrape request of your current plan or project if you set a concurrent limit at project level

ERR::SCRAPE::UNABLE_TO_TAKE_SCREENSHOT

Unable to take screenshot

ERR::SCRAPE::UPSTREAM_TIMEOUT

The website you target made too much time to response

ERR::SCRAPE::UPSTREAM_WEBSITE_ERROR

The website you tried to scrape have configuration or malformed response

Proxy

ERR::PROXY::POOL_NOT_AVAILABLE_FOR_TARGET

The desired proxy pool is not available for the given domain - mostly well known protected domain which require at least residential networks

ERR::PROXY::POOL_NOT_FOUND

Provided Proxy Pool Name do not exists

ERR::PROXY::POOL_UNAVAILABLE_COUNTRY

Country not available for given proxy pool

ERR::PROXY::RESOURCES_SATURATION

Proxy are saturated for the desired country, you can try on other countries. They will come back as soon as possible

ERR::PROXY::TIMEOUT

Proxy connection or website was too slow and timeout

ERR::PROXY::UNAVAILABLE

Proxy is unavailable - The domain (mainly gov website) is restricted, You are using session feature and the proxy is unreachable at the moment

Throttle

ERR::THROTTLE::MAX_API_CREDIT_BUDGET_EXCEEDED

Your scrape request has been throttled. API Credit Budget reached. If it's not expected, please check your throttle configuration for the given project and env.

ERR::THROTTLE::MAX_CONCURRENT_REQUEST_EXCEEDED

Your scrape request has been throttled. Too many concurrent access to the upstream. If it's not expected, please check your throttle configuration for the given project and env.

ERR::THROTTLE::MAX_REQUEST_RATE_EXCEEDED

Your scrape request as been throttle. Too much request during the 1m window. If it's not expected, please check your throttle configuration for the given project and env

Anti Scraping Protection (ASP)

ERR::ASP::CAPTCHA_ERROR

Something wrong happened with the captcha. We will figure out to fix the problem as soon as possible

ERR::ASP::CAPTCHA_TIMEOUT

The budgeted time to solve the captcha is reached

ERR::ASP::SHIELD_ERROR

The ASP encounter an unexpected problem. We will fix it as soon as possible. Our team has been alerted

ERR::ASP::SHIELD_EXPIRED

The ASP shield previously set is expired, you must retry.

  • Retryable: Yes
  • HTTP status code: 422

ERR::ASP::SHIELD_NOT_ELIGIBLE

The feature requested is not eligible while using the ASP for the given protection/target

ERR::ASP::SHIELD_PROTECTION_FAILED

The ASP shield failed to solve the challenge against the anti scrapping protection

ERR::ASP::TIMEOUT

The ASP made too much time to solve or respond

ERR::ASP::UNABLE_TO_SOLVE_CAPTCHA

Despite our effort, we were unable to solve the captcha. It can happened sporadically, please retry

ERR::ASP::UPSTREAM_UNEXPECTED_RESPONSE

The response given by the upstream after challenge resolution is not expected. Our team has been alerted

Schedule

ERR::SCHEDULE::DISABLED

The targeted schedule has been disabled

Webhook

ERR::WEBHOOK::DISABLED

Given webhook is disabled, please check out your webhook configuration for the current project / env

ERR::WEBHOOK::ENDPOINT_UNREACHABLE

We were not able to contact your endpoint

ERR::WEBHOOK::QUEUE_FULL

You reach the maximum concurrency limit

ERR::WEBHOOK::MAX_RETRY

Maximum retry exceeded on your webhook

ERR::WEBHOOK::NOT_FOUND

Unable to find the given webhook for the current project / env

ERR::WEBHOOK::QUEUE_FULL

You reach the limit of scheduled webhook - You must wait pending webhook are processed

Session

ERR::SESSION::CONCURRENT_ACCESS

Concurrent access to the session has been tried. If your spider run on distributed architecture, the same session name is currently used by another scrape

Exportable Definition

If you want to handle errors from your application without copy-pasting the whole error definition into your application to match errors, here is a portable JSON of error definition:

Summary