API Errors

Introduction

If you want to port those definitions into your application, you can checkout the Exportable Definition section to retrieve the JSON describing errors.

Generic API Error

Example of API error response :

{
    status: "error",
    http_code: 401,
    reason: "Unauthorized",
    error_id: "301e2d9e-b4f5-4289-85ea-e452143338df",
    message: "Invalid API key"
}

HTTP 400

Bad Request: Parameters sent to the API are incorrect. Check out the related documentation to figure out the error.

HTTP 404

Not Found: API URL might have a typo or be incorrect. Check out the related documentation to figure out the error.

HTTP 422

Unprocessable Entity: Your request was correct, but for some reason, we cannot handle it. Most of the time, the entity which you want to update/delete has already been processed.

HTTP 429

Too Many Request: API endpoint, which should be called with a high frequency, are throttled internally to prevent service disruption. If this happened too many times, your account would be suspended.

HTTP 500

Internal Server Error: Scrapfly is in trouble, and we have been alerted of the issue.

HTTP 502

Web service that exposes Scrapfly to the internet is in trouble, and we have been alerted of the issue.

HTTP 503

Service Temporary Unavailable: Scrapfly might run in degraded mode, or maintenance was scheduled to upgrade our service.

HTTP 504

Scrapfly is not reachable or takes too much time to respond.

Web Scraping API Errors

Scrapfly provides API errors in two locations - in the response body and the response headers:

The error details are present in the response body result.error key.

The error details are also present in the response x-scrapfly-reject-* headers.

Error Definitions

The protocol is not supported only http:// or https:// are supported


The website you target respond with an unexpected status code (>400)


Cost budget has been reached, you must increase the budget to pass this target


The DNS of the targeted website is not resolving or not responding


The Domain targeted is not allowed or restricted


The DOM Selector is invalid


The requested DOM selected is invisible (Mostly issued when element is targeted for screenshot)


The requested DOM selected was not found in rendered content within 15s


Driver used to perform the scrape can crash for many reason


Driver do not have enough resource to render the page correctly


Driver timeout - No response received


The javascript to execute goes wrong, please read the associated message to figure out the problem


Network error happened between Scrapfly server and remote server


Server of upstream website closed unexpectedly the connection


No browser available in the pool


This is a generic error for when timeout occur. It happened when internal operation took too much time


The limit set to the current project has been reached


You reach your scrape quota plan for the month. You can upgrade your plan if you want increase the quota


Submitted scenario would require more than 30s to complete


Javascript Scenario Failed


Javascript Scenario Timeout


ScrapeEngineError due to SSL, mostly due to handshake error


You reach concurrent limit of scrape request of your current plan


Unable to take screenshot, happened when renderer encounter bad formatted code


The website you target made too much time to response


The website you tried to scrape have configuration or malformed response


Proxy

The desired proxy pool is not available for the given domain - mostly well known protected domain which require at least residential networks


Provided Proxy Pool Name do not exists


Country not available for given proxy pool


Proxy are saturated for the desired country, you can try on other countries. They will come back as soon as possible


Proxy connection or website was too slow and timeout


Proxy is unavailable - The domain (mainly gov website) is restricted, You are using session feature and the proxy is unreachable at the moment


Throttle

Your scrape request has been throttled. API Credit Budget reached. If it's not expected, please check your throttle configuration for the given project and env.


Your scrape request has been throttled. Too many concurrent access to the upstream. If it's not expected, please check your throttle configuration for the given project and env.


Your scrape request as been throttle. Too much request during the 1m window. If it's not expected, please check your throttle configuration for the given project and env


Anti Scraping Protection (ASP)

Something wrong happened with the captcha. We will figure out to fix the problem as soon as possible


The budgeted time to solve the captcha is reached


The ASP encounter an unexpected problem. We will fix it as soon as possible. Our team has been alerted


The ASP shield previously set is expired, you must retry.

  • Retryable: Yes
  • HTTP status code: 422

The ASP shield failed to solve the challenge against the anti scrapping protection


The ASP made too much time to solve or respond


Despite our effort, we were unable to solve the captcha. It can happened sporadically, please retry


The response given by the upstream after challenge resolution is not expected. Our team has been alerted


Schedule

The targeted schedule has been disabled


Webhook

Given webhook is disabled, please check out your webhook configuration for the current project / env


We were not able to contact your endpoint


You reach the maximum concurrency limit


Maximum retry exceeded on your webhook


Unable to find the given webhook for the current project / env


You reach the limit of scheduled webhook - You must wait pending webhook are processed

Session

Concurrent access to the session has been tried. If your spider run on distributed architecture, check if the correlation id is correctly configured


Exportable Definition

If you want to handle errors from your application without copy-pasting the whole error definition into your application to match errors, here is a portable JSON of error definition:

Summary