API Errors

Introduction

If you want port those definitions into your application, you can checkout Exportable Definition section to retrieve the json describing errors.

Only successful request are counted into your quota. Each time a bad status code from upstream website or error is raised, you do not pay for it.

Generic API Error

Example of API error response :

openapi openapi Generic API Error | Json 
            {
    status: "error",
    http_code: 401,
    reason: "Unauthorized",
    error_id: "301e2d9e-b4f5-4289-85ea-e452143338df",
    message: "Invalid API key"
}

HTTP 400

Bad Request - Parameters sent to the API are incorrect. Checkout the related documentation to figured out

HTTP 404

Not Found - API url might have typo or incorrect. Checkout the related documentation to figured out

HTTP 422

Unprocessable Entity - Your request was correct but for some reason we cannot handle it. Most of the time, the entity which you want update / delete has already been processed

HTTP 429

Too Many Request - API endpoint which should be called with high frequency are throttled internally to prevent service disruption. If it happened too much time, your account might be suspended

HTTP 500

Internal Server Error - Scrapfly is in trouble and we have been alerted of the issue

HTTP 502

Web service that expose Scrapfly to internet is in trouble and we have been alerted of the issue

HTTP 503

Service Temporary Unavailable - Scrapfly might run in degraded mode or a maintenance was scheduled to upgrade our service

HTTP 504

Scrapfly is not reachable or make too much time to respond.

Scrape

Example of Scrape API response:

Scrape API Error Structure

openapi openapi Scrape API Error | Json 
            {
    "config": { ... },
    "context": { ... },
    "result": {
        [...],
        "status": "DONE",
        "success": false,
        "reason": null,
        "error": {
            "http_code": 429,
            "code": "ERR::THROTTLE::MAX_REQUEST_RATE_EXCEEDED",
            "description": "Your scrape request as been throttle. Too much request during the last minute. If it's not expected, please check your throttle configuration for the given project and env",
            "error_id": "9993a546-b899-4927-b788-04f5c4e473d5",
            "message": "Max request rate exceeded",
            "scrape_id": "7c61352c-f1a7-4ea6-a0b8-198d7ac6fe1a",
            "retryable": false,
            "doc_url": "https://scrapfly.local/docs/scrape-api/error/ERR::THROTTLE::MAX_REQUEST_RATE_EXCEEDED"
        },
        [...],
    }
}

ERR::SCRAPE::QUOTA_LIMIT_REACHED

You reach your scrape quota plan for the month. You can upgrade your plan if you want increase the quota

  • Retryable: No
  • HTTP status code: 429

ERR::SCRAPE::OPERATION_TIMEOUT

This is a generic error for when timeout occur. It happened when internal operation took too much time

  • Retryable: Yes
  • HTTP status code: 408

ERR::SCRAPE::PROJECT_QUOTA_LIMIT_REACHED

The limit set to the current project has been reached

  • Retryable: No
  • HTTP status code: 429

ERR::SCRAPE::TOO_MANY_CONCURRENT_REQUEST

You reach concurrent limit of scrape request of your current plan

  • Retryable: Yes
  • HTTP status code: 429

ERR::SCRAPE::UNABLE_TO_TAKE_SCREENSHOT

Unable to take screenshot, happened when renderer encounter bad formatted code

  • Retryable: Yes
  • HTTP status code: 520

ERR::SCRAPE::UPSTREAM_TIMEOUT

Upstream website made too much time to response

  • Retryable: No
  • HTTP status code: 408

ERR::SCRAPE::JAVASCRIPT_EXECUTION

The javascript to execute goes wrong, please read the associated message to figure out the problem

  • Retryable: No
  • HTTP status code: 500
Related Documentation:

ERR::SCRAPE::GRAPHQL_QUERY_ERROR

Bad GraphQL Query, please read the associated message to figure out what wrong

  • Retryable: No
  • HTTP status code: 400
Related Documentation:

ERR::SCRAPE::BAD_UPSTREAM_RESPONSE

Upstream respond with http code >= 299

HTTP Status code relate the state of the service. Forwarding you the upstream response code as our api response code is a non sense, it will create confusion & complexity between regular Scrapfly error and do not represent the correct current state of our service. For simplicity of client-side implementation, we decided to treat bad upstream response code as regular error with same structure and answer with a 200. Not conventional but we think it's the best solution
  • Retryable: No
  • HTTP status code: 200

ERR::SCRAPE::GRAPHQL_UNEXPECTED_ERROR

Something goes wrong for an unexpected reason, our team has been alerted

  • Retryable: No
  • HTTP status code: 500
Related Documentation:

ERR::SCRAPE::GRAPHQL_INVALID_QUERY

Semantic of the query is incorrect, please the associated message to figure out the problem

  • Retryable: No
  • HTTP status code: 400
Related Documentation:

ERR::SCRAPE::DNS_NAME_NOT_RESOLVED

The DNS of upstream website is not resolving or not responding

  • Retryable: No
  • HTTP status code: 523
Related Documentation:

ERR::SCRAPE::BAD_PROTOCOL

The protocol is not supported

  • Retryable: No
  • HTTP status code: 500

ERR::SCRAPE::NETWORK_SERVER_DISCONNECTED

Server of upstream website closed unexpectedly the connection

  • Retryable: No
  • HTTP status code: 499

ERR::SCRAPE::NETWORK_ERROR

Network error happened between Scrapfly server and remote server

This king of error are automatically retried internally if auto retry is not disabled. If you see this error, it means it already has been tried 5 times
  • Retryable: Yes
  • HTTP status code: 499

ERR::SCRAPE::DRIVER_CRASHED

Driver used to perform the scrape can crash for many reason

  • Retryable: Yes
  • HTTP status code: 503

Proxy

ERR::PROXY::RESOURCES_SATURATION

Proxy are saturated for the desired country, you can try on other countries. They will come back as soon as possible

  • Retryable: Yes
  • HTTP status code: 503

ERR::PROXY::NOT_REACHABLE

Proxy was not reachable, it can happened when network issue or proxy itself is in trouble

  • Retryable: Yes
  • HTTP status code: 503

ERR::PROXY::NETWORK_DOWN

Proxy network is down

  • Retryable: No
  • HTTP status code: 503

ERR::PROXY::POOL_NOT_FOUND

Provided Proxy Pool Name do not exists

  • Retryable: No
  • HTTP status code: 400
Related Documentation:

Throttle

ERR::THROTTLE::MAX_CONCURRENT_REQUEST_EXCEEDED

Your scrape request has been throttled. Too many concurrent access to the upstream. If it's not expected, please check your throttle configuration for the given project and env.

  • Retryable: Yes
  • HTTP status code: 429

ERR::THROTTLE::MAX_REQUEST_RATE_EXCEEDED

Your scrape request as been throttle. Too much request during the last minute. If it's not expected, please check your throttle configuration for the given project and env

  • Retryable: Yes
  • HTTP status code: 429

Anti Scraping Protection (ASP)

ERR::ASP::PROTECTION_FAILED

Something wrong happened with specific shield of anti bot protection. We will figure out to fix the problem as soon as possible. Sometimes Anti bot protection change their technics and we need to adapt our system

  • Retryable: No
  • HTTP status code: 520

ERR::ASP::CAPTCHA_ERROR

Something wrong happened with the captcha. We will figure out to fix the problem as soon as possible

  • Retryable: No
  • HTTP status code: 500

ERR::ASP::UNABLE_TO_SOLVE_CAPTCHA

Despite our effort, we were unable to solve the captcha. It can happened sporadically, please retry

  • Retryable: Yes
  • HTTP status code: 520

ERR::ASP::CAPTCHA_TIMEOUT

The budgeted time to solve the captcha is reached

  • Retryable: Yes
  • HTTP status code: 408

ERR::ASP::UPSTREAM_UNEXPECTED_RESPONSE

The response given by the upstream after challenge resolution is not expected. Our team has been alerted

  • Retryable: No
  • HTTP status code: 520

ERR::ASP::SHIELD_ERROR

The ASP encounter an unexpected problem. We will fix it as soon as possible. Our team has been alerted

  • Retryable: No
  • HTTP status code: 520

ERR::ASP::SHIELD_EXPIRED

The ASP shield previously set is expired, you must retry.

This issue should not be raised on your side, it's managed internally.
  • Retryable: Yes
  • HTTP status code: 419

ERR::ASP::SHIELD_FAILED

The ASP shield failed to solve the challenge against the anti scrapping protection

  • Retryable: Yes
  • HTTP status code: 520

ERR::ASP::TIMEOUT

The ASP made too much time to solve or respond

  • Retryable: Yes
  • HTTP status code: 408

Schedule

ERR::SCHEDULE::DISABLED

The targeted schedule has been disabled

  • Retryable: No
  • HTTP status code: 504

Webhook

ERR::WEBHOOK::NOT_FOUND

Unable to find the given webhook for the current project / env

  • Retryable: No
  • HTTP status code: 400
Related Documentation:

ERR::WEBHOOK::DISABLED

Given webhook is disabled, please check out your webhook configuration for the current project / env

  • Retryable: No
  • HTTP status code: 500
Related Documentation:

ERR::WEBHOOK::MAX_RETRY

Maximum retry exceeded on your webhook

  • Retryable: No
  • HTTP status code: 429
Related Documentation:

Session

ERR::SESSION::CONCURRENT_ACCESS

Concurrent access to the session has been tried. If your spider run on distributed architecture, check if the correlation id is correctly configured

  • Retryable: Yes
  • HTTP status code: 429
Related Documentation:

Exportable Definition

If you want handle errors from your application without copy pasting the whole error definition into your application to match errors, here is a portable json of error definition :

openapi openapi Errors definitions | Json 
            {
    "ERR::PROXY::RESOURCES_SATURATION": {
        "description": "Proxy are saturated for the desired country, you can try on other countries. They will come back as soon as possible",
        "retryable": true,
        "response_code": 503,
        "docs": [],
        "code": "ERR::PROXY::RESOURCES_SATURATION"
    },
    "ERR::PROXY::NOT_REACHABLE": {
        "description": "Proxy was not reachable, it can happened when network issue or proxy itself is in trouble",
        "retryable": true,
        "response_code": 503,
        "docs": [],
        "code": "ERR::PROXY::NOT_REACHABLE"
    },
    "ERR::PROXY::NETWORK_DOWN": {
        "description": "Proxy network is down",
        "retryable": false,
        "response_code": 503,
        "docs": [],
        "code": "ERR::PROXY::NETWORK_DOWN"
    },
    "ERR::PROXY::POOL_NOT_FOUND": {
        "description": "Provided Proxy Pool Name do not exists",
        "retryable": false,
        "response_code": 400,
        "docs": {
            "Checkout Proxy Documentation": "https://scrapfly.io/docs/scrape-api/proxy"
        },
        "code": "ERR::PROXY::POOL_NOT_FOUND"
    },
    "ERR::THROTTLE::MAX_CONCURRENT_REQUEST_EXCEEDED": {
        "description": "Your scrape request has been throttled. Too many concurrent access to the upstream. If it's not expected, please check your throttle configuration for the given project and env.",
        "retryable": true,
        "response_code": 429,
        "docs": [],
        "code": "ERR::THROTTLE::MAX_CONCURRENT_REQUEST_EXCEEDED"
    },
    "ERR::THROTTLE::MAX_REQUEST_RATE_EXCEEDED": {
        "description": "Your scrape request as been throttle. Too much request during the last minute. If it's not expected, please check your throttle configuration for the given project and env",
        "retryable": true,
        "response_code": 429,
        "docs": [],
        "code": "ERR::THROTTLE::MAX_REQUEST_RATE_EXCEEDED"
    },
    "ERR::ASP::PROTECTION_FAILED": {
        "description": "Something wrong happened with specific shield of anti bot protection. We will figure out to fix the problem as soon as possible. Sometimes Anti bot protection change their technics and we need to adapt our system",
        "retryable": false,
        "response_code": 520,
        "docs": [],
        "code": "ERR::ASP::PROTECTION_FAILED"
    },
    "ERR::ASP::CAPTCHA_ERROR": {
        "description": "Something wrong happened with the captcha. We will figure out to fix the problem as soon as possible",
        "retryable": false,
        "response_code": 500,
        "docs": [],
        "code": "ERR::ASP::CAPTCHA_ERROR"
    },
    "ERR::ASP::UNABLE_TO_SOLVE_CAPTCHA": {
        "description": "Despite our effort, we were unable to solve the captcha. It can happened sporadically, please retry",
        "retryable": true,
        "response_code": 520,
        "docs": [],
        "code": "ERR::ASP::UNABLE_TO_SOLVE_CAPTCHA"
    },
    "ERR::ASP::CAPTCHA_TIMEOUT": {
        "description": "The budgeted time to solve the captcha is reached",
        "retryable": true,
        "response_code": 408,
        "docs": [],
        "code": "ERR::ASP::CAPTCHA_TIMEOUT"
    },
    "ERR::ASP::UPSTREAM_UNEXPECTED_RESPONSE": {
        "description": "The response given by the upstream after challenge resolution is not expected. Our team has been alerted",
        "retryable": false,
        "response_code": 520,
        "docs": [],
        "code": "ERR::ASP::UPSTREAM_UNEXPECTED_RESPONSE"
    },
    "ERR::ASP::SHIELD_ERROR": {
        "description": "The ASP encounter an unexpected problem. We will fix it as soon as possible. Our team has been alerted",
        "retryable": false,
        "response_code": 520,
        "docs": [],
        "code": "ERR::ASP::SHIELD_ERROR"
    },
    "ERR::ASP::SHIELD_EXPIRED": {
        "description": "The ASP shield previously set is expired, you must retry.",
        "context": "This issue should not be raised on your side, it's managed internally.",
        "retryable": true,
        "response_code": 419,
        "docs": [],
        "code": "ERR::ASP::SHIELD_EXPIRED"
    },
    "ERR::ASP::SHIELD_FAILED": {
        "description": "The ASP shield failed to solve the challenge against the anti scrapping protection",
        "retryable": true,
        "response_code": 520,
        "docs": [],
        "code": "ERR::ASP::SHIELD_FAILED"
    },
    "ERR::ASP::TIMEOUT": {
        "description": "The ASP made too much time to solve or respond",
        "retryable": true,
        "response_code": 408,
        "docs": [],
        "code": "ERR::ASP::TIMEOUT"
    },
    "ERR::SCHEDULE::DISABLED": {
        "description": "The targeted schedule has been disabled",
        "retryable": false,
        "response_code": 504,
        "docs": [],
        "code": "ERR::SCHEDULE::DISABLED"
    },
    "ERR::SCRAPE::QUOTA_LIMIT_REACHED": {
        "description": "You reach your scrape quota plan for the month. You can upgrade your plan if you want increase the quota",
        "retryable": false,
        "response_code": 429,
        "docs": [],
        "code": "ERR::SCRAPE::QUOTA_LIMIT_REACHED"
    },
    "ERR::SCRAPE::OPERATION_TIMEOUT": {
        "description": "This is a generic error for when timeout occur. It happened when internal operation took too much time",
        "retryable": true,
        "response_code": 408,
        "docs": [],
        "code": "ERR::SCRAPE::OPERATION_TIMEOUT"
    },
    "ERR::SCRAPE::PROJECT_QUOTA_LIMIT_REACHED": {
        "description": "The limit set to the current project has been reached",
        "retryable": false,
        "response_code": 429,
        "docs": [],
        "code": "ERR::SCRAPE::PROJECT_QUOTA_LIMIT_REACHED"
    },
    "ERR::SCRAPE::TOO_MANY_CONCURRENT_REQUEST": {
        "description": "You reach concurrent limit of scrape request of your current plan",
        "retryable": true,
        "response_code": 429,
        "docs": [],
        "code": "ERR::SCRAPE::TOO_MANY_CONCURRENT_REQUEST"
    },
    "ERR::SCRAPE::UNABLE_TO_TAKE_SCREENSHOT": {
        "description": "Unable to take screenshot, happened when renderer encounter bad formatted code",
        "retryable": true,
        "response_code": 520,
        "docs": [],
        "code": "ERR::SCRAPE::UNABLE_TO_TAKE_SCREENSHOT"
    },
    "ERR::SCRAPE::UPSTREAM_TIMEOUT": {
        "description": "Upstream website made too much time to response",
        "retryable": false,
        "response_code": 408,
        "docs": [],
        "code": "ERR::SCRAPE::UPSTREAM_TIMEOUT"
    },
    "ERR::SCRAPE::JAVASCRIPT_EXECUTION": {
        "description": "The javascript to execute goes wrong, please read the associated message to figure out the problem",
        "retryable": false,
        "response_code": 500,
        "docs": {
            "Checkout Javascript Rendering Documentation": "https://scrapfly.io/docs/scrape-api/javascript-rendering"
        },
        "code": "ERR::SCRAPE::JAVASCRIPT_EXECUTION"
    },
    "ERR::SCRAPE::GRAPHQL_QUERY_ERROR": {
        "description": "Bad GraphQL Query, please read the associated message to figure out what wrong",
        "retryable": false,
        "response_code": 400,
        "docs": {
            "Checkout GraphQL Documentation": "https://scrapfly.io/docs/scrape-api/graphql"
        },
        "code": "ERR::SCRAPE::GRAPHQL_QUERY_ERROR"
    },
    "ERR::SCRAPE::BAD_UPSTREAM_RESPONSE": {
        "description": "Upstream respond with http code >= 299",
        "retryable": false,
        "context": "HTTP Status code relate the state of the service. Forwarding you the upstream response code as our api response code is a non sense, it will create confusion & complexity between regular Scrapfly error and do not represent the correct current state of our service. For simplicity of client-side implementation, we decided to treat bad upstream response code as regular error with same structure and answer with a 200. Not conventional but we think it's the best solution",
        "response_code": 200,
        "docs": [],
        "code": "ERR::SCRAPE::BAD_UPSTREAM_RESPONSE"
    },
    "ERR::SCRAPE::GRAPHQL_UNEXPECTED_ERROR": {
        "description": "Something goes wrong for an unexpected reason, our team has been alerted",
        "retryable": false,
        "response_code": 500,
        "docs": {
            "Checkout GraphQL Documentation": "https://scrapfly.io/docs/scrape-api/graphql"
        },
        "code": "ERR::SCRAPE::GRAPHQL_UNEXPECTED_ERROR"
    },
    "ERR::SCRAPE::GRAPHQL_INVALID_QUERY": {
        "description": "Semantic of the query is incorrect, please the associated message to figure out the problem",
        "retryable": false,
        "response_code": 400,
        "docs": {
            "Checkout GraphQL Documentation": "https://scrapfly.io/docs/scrape-api/graphql"
        },
        "code": "ERR::SCRAPE::GRAPHQL_INVALID_QUERY"
    },
    "ERR::SCRAPE::DNS_NAME_NOT_RESOLVED": {
        "description": "The DNS of upstream website is not resolving or not responding",
        "retryable": false,
        "response_code": 523,
        "docs": {
            "Checkout GraphQL Documentation": "https://scrapfly.io/docs/scrape-api/graphql"
        },
        "code": "ERR::SCRAPE::DNS_NAME_NOT_RESOLVED"
    },
    "ERR::SCRAPE::BAD_PROTOCOL": {
        "description": "The protocol is not supported",
        "retryable": false,
        "response_code": 500,
        "docs": [],
        "code": "ERR::SCRAPE::BAD_PROTOCOL"
    },
    "ERR::SCRAPE::NETWORK_SERVER_DISCONNECTED": {
        "description": "Server of upstream website closed unexpectedly the connection",
        "retryable": false,
        "response_code": 499,
        "docs": [],
        "code": "ERR::SCRAPE::NETWORK_SERVER_DISCONNECTED"
    },
    "ERR::SCRAPE::NETWORK_ERROR": {
        "description": "Network error happened between Scrapfly server and remote server",
        "context": "This king of error are automatically retried internally if auto retry is not disabled. If you see this error, it means it already has been tried 5 times",
        "retryable": true,
        "response_code": 499,
        "docs": [],
        "code": "ERR::SCRAPE::NETWORK_ERROR"
    },
    "ERR::SCRAPE::DRIVER_CRASHED": {
        "description": "Driver used to perform the scrape can crash for many reason",
        "retryable": true,
        "response_code": 503,
        "docs": [],
        "code": "ERR::SCRAPE::DRIVER_CRASHED"
    },
    "ERR::WEBHOOK::NOT_FOUND": {
        "description": "Unable to find the given webhook for the current project / env",
        "retryable": false,
        "response_code": 400,
        "docs": {
            "Checkout Webhook Documentation": "https://scrapfly.io/docs/scrape-api/screenshot"
        },
        "code": "ERR::WEBHOOK::NOT_FOUND"
    },
    "ERR::WEBHOOK::DISABLED": {
        "description": "Given webhook is disabled, please check out your webhook configuration for the current project / env",
        "retryable": false,
        "response_code": 500,
        "docs": {
            "Checkout Webhook Documentation": "https://scrapfly.io/docs/scrape-api/screenshot"
        },
        "code": "ERR::WEBHOOK::DISABLED"
    },
    "ERR::WEBHOOK::MAX_RETRY": {
        "description": "Maximum retry exceeded on your webhook",
        "retryable": false,
        "response_code": 429,
        "docs": {
            "Checkout Webhook Documentation": "https://scrapfly.io/docs/scrape-api/screenshot"
        },
        "code": "ERR::WEBHOOK::MAX_RETRY"
    },
    "ERR::SESSION::CONCURRENT_ACCESS": {
        "description": "Concurrent access to the session has been tried. If your spider run on distributed architecture, check if the correlation id is correctly configured",
        "retryable": true,
        "response_code": 429,
        "docs": {
            "Checkout Session Documentation": "https://scrapfly.io/docs/scrape-api/session"
        },
        "code": "ERR::SESSION::CONCURRENT_ACCESS"
    }
}