Web Scraper API Specification
Scrapfly has loads of features and the best way to discover them is through the specification docs below. Alternatively, see the OpenAPI specification if you're familiar with openAPI structures.
If you have any questions you can check out the Frequently asked question section or see the support chat.
By default, the API has a read timeout of 155 seconds. To avoid read timeout errors, you must configure your HTTP client to set the read timeout to 155 seconds. If you need a different timeout value, please refer to the documentation for information on how to control the timeout.
Try out the API directly in your terminal using curl:
Want to try out the API without coding? Check out our visual API player and test/generate code to use our API.
The default response format is JSON, and the scraped content is available in
result.content. Your scrape configuration is present in
config, and other activated feature information is available in
context.
To get the HTML page directly, refer to the
proxified_response parameter.
HTTP Parameter
Description
Example
Target url to scrape. Must be url encoded
url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this
Select the proxy pool. A proxy pool is a network of proxy grouped by quality range and network type. The price vary base on the pool used.
proxy_pool=public_datacenter_pool
proxy_pool=public_residential_pool
Pass custom headers to request. To learn about request customization, you cache checkout the
dedicated documentation
Must be url encoded
headers[content-type]=application%2Fjson
headers[Cookie]=test%3D1%3Bauth%3D1
Proxy country location - If not set it chooses a random location available.
A reference to a country must be ISO 3166 alpha-2 (2 letters).
The available countries are defined by the proxy pool you use.
-
country=us -
country=us,ca,mx -
country=us:1,ca:5,mx:3,-gb -
country=-gb
Select desired lang - By default it use the language from the proxy location. Behind the scene it configures the
Accept-Language
HTTP header.
If the website support the language, the content will be in that lang.
You can't set lang parameter and
Accept-Language header
-
lang=en -
lang=ch-FR,fr-FR,en -
lang=en-IN,en-US
Operating System, if not selected it's random.
You can't set os parameter and
User-Agent
header
-
os=win11 -
os=mac -
os=linux -
os=chromeos
Timeout is expressed in milliseconds. It represents the maximum time allowed to Scrapfly to try to perform the scrape. Since
timeout
is
not trivial to understand regarding other settings -
a full documentation is available
-
timeout=30000 -
timeout=120000
Improve reliability, retry on failure (network, upstream http code >= 500). When internal retry is enabled.
Retry parameter have an impact on
timeout.
-
retry=true -
retry=false
By default API respond with a JSON response and the content is located in
result.content. With
proxified_response
the content of the page is directly returned as body and status code / headers are replaced by the
target response.
-
proxified_response=true -
proxified_response=false
Store the API result and take a screenshot if the rendering js is enabled. A sharable link with the saved response is available.
It's usefull to enable when you have an issue to communicate with our support.
debug=true
debug=false
Help to correlate a group of scrape issued by the same worker or machine. You can use it as filter in our monitoring dashboard
correlation_id=e3ba784cde0d
Add tags to your calls to easily group or filter them with many values.
tags[]=jewelery
tags[]=price
Pull remote ssl certificate and return other tls information. Only available for
https://.
You do not need to enable it for scraping
https://
target - it works by default, it just add more
information.
ssl=true
ssl=false
Queue you scrape request and received the API response on your webhook endpoint. You can create a webhook endpoint from your
dashboard, it takes the name of the webhook. Webhooks are scoped to the given project/env.
webhook_name=my-webhook-name
Anti Scraping Protection
All related parameters require
asp
enabled
Anti Scraping Protection
- Unblock protected website and bypass protection
asp=true
asp=false
ASP
dynamically retry and upgrade some parameters (such as proxy_pool, browser) to pass and this change dynamically the cost of the call,
to make it more predictable, you can define a budget to respect. Make sure to set the minimum required to pass your target or the call will be rejected without even trying.
cost_budget=25
cost_budget=55
Headless Browser / Javascript Rendering
All related parameters require
render_js
enabled
Enable browser rendering. Scrape the target with a browser and render the page
-
render_js=true -
render_js=false
Delay in milli seconds to wait after the page was loaded.
-
rendering_wait=5000
Await until given CSS Selector or XPATH is visible in the page
-
wait_for_selector=body -
wait_for_selector=input[type="submit"] -
wait_for_selector=//button[contains(text(),"Go")]
Script to execute in the page.
Must be base64 encoded with url safe option.
-
js=Y29uc29sZS5sb2coJ3Rlc3QnKQ
Take screenshot - You can take multiple screenshot with diffrent format: fullpage or specific element selected via CSS Selector or XPATH.
The key argument is the name of the screenshot and the value is the selector.
-
screenshots[page]=fullpagescreenshots[price]=#price
User scenario to navigate or perform action on the page. It's a JSON and
must be base64 encoded with url safe option.
-
js_scenario=eydjbGljayc6IHsnc2VsZWN0b3InOiAnI3N1Ym1pdCd9fQ
Grant geolocation permission and spoof latitude and longitude. Format:
latitude,longitude
-
geolocation=48.856614,2.3522219 -
geolocation=-74.005941,40.712784
Auto Scroll to the bottom of the page. It allows to trigger javascript rendering based on the user viewport intersection.
-
auto_scroll=true -
auto_scroll=false
Stage to wait when rendering the page, You can choose between
complete which is the default, or domcontentloaded if you want a fast render
without waiting the full rendering (faster scrape)
-
rendering_stage=completerendering_stage=domcontentloaded
Cache
All related parameters require
cache
enabled
Enable the cache layer. If the cache is MISS the scrape is performed otherwise the cached content is returned. If the TTL is expired, the
cache will be refreshed by scraping again the target.
-
cache=true -
cache=false
Cache Time To Live in seconds. If you do a scrape after the TTL expired, the target will be scraped and the cache refresh with the new data.
-
cache_ttl=60 -
cache_ttl=3600
Force to refresh the cache and ensure the scrape is performed to cache the new version
-
cache_clear=true -
cache_clear=false
Session
All related parameters require
session
enabled
Session name to be reused - Automatically store and restore cookies, fingerprint and proxy across many scrapes. Must be alphanumeric, max length 255 chars
-
session=17013313
Best effort is made to reuse the same proxy ip
-
session_sticky_proxy=true -
session_sticky_proxy=false