Frequently Asked Question

How to send a POST request

We have a dedicated documentation to explain it

How to send cookies or specific headers

We have a dedicated documentation to explain it

How to know the API credit billed and the detailed

If you directly want the total of API credit billed, you can check out the header X-Scrapfly-Api-Cost. If you want to get the details, you have the information in our JSON response response.context.cost where you can find the detail and the total.

How to check the concurrent usage

API Response contains the header X-Scrapfly-Account-Remaining-Concurrent-Usage which indicate the remaining concurrent slot available before getting 429 - ERR::SCRAPE::TOO_MANY_CONCURRENT_REQUEST As soon as a slot is available (a scrape is terminated) you will be able to scrape again. If you want to know the global usage of the account, you can check X-Scrapfly-Account-Concurrent-Usage. If your project have a quota to limit the concurrency, you can check the header X-Scrapfly-Project-Remaining-Concurrent-Usage which indicate the remaining concurrent slot available for the current project (each project have it own api key). If you want to know the current usage X-Scrapfly-Project-Concurrent-Usage

How prevent extra usage

API Response contains the header X-Scrapfly-Remaining-Api-Credit which indicates you the amount of API count on your account. If the value is 0 then you are in extra usage. You can also have account information (quota, concurrency and so on) via our Account API

How bypass protected website / unblock my scrape

You can check out the ASP feature and the API parameter to use it.

How get the scraped result directly

Yes, you can set proxified_response=true as url parameter. Body and Headers are from upstream. You can check out directly the parameter documentation

How get the page correctly rendered

If your page do not render what you expect here is the guideline to troubleshoot correctly :

  • In your browser CTRL+u to see how the page is without the rendered javascript. If the expected content is not inside, you need javascript rendering
  • If the element take time to be rendered, you can try to setup rendering_wait or wait_for_selector
  • Make sure page the page you scrape is not blocked - check out ASP feature to unblock it

How to download image or pdf

Yes, you can check the format of content via response['result']['format'], text or binary. If the content is a binary format, the content is base64 encoded. You must not use browser rendering when you want to download media/image, most of the time, regarding the content-type, the browser will load by generating html document and load it through media html tag (img, video, audio)

Request with ASP are too expensive

We know it's frustrating to scrape protected website - but it also had a cost to bypass most of the time they require residential proxies and browser to pass challenge. Protection evolves and need to be updated - we have reverse engineer team dedicated to the that.

I'm Getting Read Timeout Error

The API read timeout is 155s by default. You must configure your http client to set the read timeout to 155. If you don't want this value and want to avoid Read timeout error, you must set retry=false.

How API Credit does it cost

  • By default (Datacenter without browser rendering): 1 API Credit
  • With Browser Rendering: +5 API Credit
  • With Residential Proxy Network: +25 API Credit
  • Datacenter Proxy Network + Browser Rendering: 6 API Credit
  • Residential Proxy Network + Browser Rendering: 30 API Credit
  • Some domain can cost some extra API Credit - You can contact the support or try your target in the API Player

You can checkout the dedicated pricing page.

The easiest way to project your cost and evaluate the budget is to test calls and reproduce it through or UI API player (no code required) and look at the cost - Each API call have it's own logs with all the details about the API Credit billed step by step.