# Scrapfly Documentation

## Table of Contents

### Dashboard

- [Intro](https://scrapfly.io/docs)
- [Project](https://scrapfly.io/docs/project)
- [Account](https://scrapfly.io/docs/account)
- [Workspace & Team](https://scrapfly.io/docs/workspace-and-team)
- [Billing](https://scrapfly.io/docs/billing)

### Products

#### MCP Server

- [Getting Started](https://scrapfly.io/docs/mcp/getting-started)
- [Tools & API Spec](https://scrapfly.io/docs/mcp/tools)
- [Authentication](https://scrapfly.io/docs/mcp/authentication)
- [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples)
- [FAQ](https://scrapfly.io/docs/mcp/faq)
##### Integrations

- [Overview](https://scrapfly.io/docs/mcp/integrations)
- [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop)
- [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code)
- [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt)
- [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor)
- [Cline](https://scrapfly.io/docs/mcp/integrations/cline)
- [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf)
- [Zed](https://scrapfly.io/docs/mcp/integrations/zed)
- [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code)
- [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode)
- [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain)
- [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex)
- [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai)
- [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai)
- [n8n](https://scrapfly.io/docs/mcp/integrations/n8n)
- [Make](https://scrapfly.io/docs/mcp/integrations/make)
- [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier)
- [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi)
- [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder)
- [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client)


#### Web Scraping API

- [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started)
- [API Specification]()
- [Monitoring](https://scrapfly.io/docs/monitoring)
- [Customize Request](https://scrapfly.io/docs/scrape-api/custom)
- [Debug](https://scrapfly.io/docs/scrape-api/debug)
- [Unblocker (formerly ASP)](https://scrapfly.io/docs/scrape-api/unblocker)
- [Proxy](https://scrapfly.io/docs/scrape-api/proxy)
- [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode)
- [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog)
- [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify)
- [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction)
- [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering)
- [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario)
- [SSL](https://scrapfly.io/docs/scrape-api/ssl)
- [DNS](https://scrapfly.io/docs/scrape-api/dns)
- [Cache](https://scrapfly.io/docs/scrape-api/cache)
- [Batch (Multi-URL Scraping)](https://scrapfly.io/docs/scrape-api/batch)
- [Session](https://scrapfly.io/docs/scrape-api/session)
- [Webhook](https://scrapfly.io/docs/scrape-api/webhook)
- [Schedule](https://scrapfly.io/docs/scrape-api/schedule)
- [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot)
- [Errors](https://scrapfly.io/docs/scrape-api/errors)
- [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout)
- [Throttling](https://scrapfly.io/docs/throttling)
- [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot)
- [Billing](https://scrapfly.io/docs/scrape-api/billing)
- [FAQ](https://scrapfly.io/docs/scrape-api/faq)

#### Crawler API

- [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started)
- [API Specification]()
- [Retrieving Results](https://scrapfly.io/docs/crawler-api/results)
- [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format)
- [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules)
- [Search](https://scrapfly.io/docs/crawler-api/search)
- [Prompt & Extract](https://scrapfly.io/docs/crawler-api/prompt)
- [Auto Refresh](https://scrapfly.io/docs/crawler-api/refresh)
- [Webhook](https://scrapfly.io/docs/crawler-api/webhook)
- [Schedule](https://scrapfly.io/docs/crawler-api/schedule)
- [Billing](https://scrapfly.io/docs/crawler-api/billing)
- [Errors](https://scrapfly.io/docs/crawler-api/errors)
- [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot)
- [FAQ](https://scrapfly.io/docs/crawler-api/faq)

#### Screenshot API

- [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started)
- [API Specification]()
- [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility)
- [Webhook](https://scrapfly.io/docs/screenshot-api/webhook)
- [Schedule](https://scrapfly.io/docs/screenshot-api/schedule)
- [Billing](https://scrapfly.io/docs/screenshot-api/billing)
- [Errors](https://scrapfly.io/docs/screenshot-api/errors)

#### Extraction API

- [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started)
- [API Specification]()
- [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template)
- [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt)
- [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai)
- [Webhook](https://scrapfly.io/docs/extraction-api/webhook)
- [Billing](https://scrapfly.io/docs/extraction-api/billing)
- [Errors](https://scrapfly.io/docs/extraction-api/errors)
- [FAQ](https://scrapfly.io/docs/extraction-api/faq)

#### Data API


#### Proxy Saver

- [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started)
- [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints)
- [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations)
- [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates)
- [Protocols](https://scrapfly.io/docs/proxy-saver/protocols)
- [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile)
- [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security)
- [Billing](https://scrapfly.io/docs/proxy-saver/billing)

#### Cloud Browser API

- [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started)
- [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy)
- [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock)
- [Captcha Solver](https://scrapfly.io/docs/cloud-browser-api/captcha-solver)
- [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads)
- [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume)
- [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop)
- [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode)
- [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions)
- [Credential Vault](https://scrapfly.io/docs/cloud-browser-api/credential-vault)
- [Native Browser MCP](https://scrapfly.io/docs/cloud-browser-api/mcp)
- [DevTools Protocol](https://scrapfly.io/docs/cloud-browser-api/cdp-reference)
##### Integrations

- [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer)
- [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright)
- [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium)
- [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser)
- [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use)
- [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand)
- [1Password](https://scrapfly.io/docs/cloud-browser-api/1password)

- [Billing](https://scrapfly.io/docs/cloud-browser-api/billing)
- [Errors](https://scrapfly.io/docs/cloud-browser-api/errors)


### Tools

- [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector)

### SDK

- [Golang](https://scrapfly.io/docs/sdk/golang)
- [PHP](https://scrapfly.io/docs/sdk/php)
- [Python](https://scrapfly.io/docs/sdk/python)
- [Rust](https://scrapfly.io/docs/sdk/rust)
- [TypeScript](https://scrapfly.io/docs/sdk/typescript)
- [Scrapy](https://scrapfly.io/docs/sdk/scrapy)

### Integrations

- [Getting Started](https://scrapfly.io/docs/integration/getting-started)
- [LangChain](https://scrapfly.io/docs/integration/langchain)
- [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex)
- [CrewAI](https://scrapfly.io/docs/integration/crewai)
- [Zapier](https://scrapfly.io/docs/integration/zapier)
- [Make](https://scrapfly.io/docs/integration/make)
- [n8n](https://scrapfly.io/docs/integration/n8n)

### Academy

- [Overview](https://scrapfly.io/academy)
- [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview)
- [Tools](https://scrapfly.io/academy/tools-overview)
- [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering)
- [Static Scraping](https://scrapfly.io/academy/static-scraping)
- [HTML Parsing](https://scrapfly.io/academy/html-parsing)
- [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping)
- [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping)
- [Headless Browsers](https://scrapfly.io/academy/headless-browsers)
- [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data)
- [JSON Parsing](https://scrapfly.io/academy/json-parsing)
- [Data Processing](https://scrapfly.io/academy/data-processing)
- [Scaling](https://scrapfly.io/academy/scaling)
- [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary)
- [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking)
- [Proxies](https://scrapfly.io/academy/proxies)

---

# Getting Started with Scrapfly

 Discover how to use Scrapfly API - the basics, available parameters and features, error handling and other information related to the API use.

 Minimal API call is a GET, POST, PUT, PATCH or HEAD request with `url` and `key` parameters:

```
https://api.scrapfly.io/scrape?url=<URL>&key=
```

 [ See Usage ](#spec)

## On Steroids

- Smart defaults - **scrape without being blocked** . Scrapfly pre-configures user-agent and other request headers.
- The [Unblocker](https://scrapfly.io/docs/scrape-api/unblocker) feature bypasses all anti-scraping systems.
- By default, the API responds in JSON. Though, a more efficient `msgpack` format is also available by setting the accept: application/msgpack header.
- Text content is returned as utf-8 while binary is encoded in base64, so you can scrape any kind of data (pdf, zip, etc)
- Gzip compression is available through content-encoding: gzip header.
- Ability to [debug and replay scrape requests](https://scrapfly.io/docs/scrape-api/debug) from the dashboard log page and API.
- [Handle large payload](#handle_large_object), large text response greater than 5MB are called "CLOB" (Character Large Object) and binary are called "BLOB" (Binary Large Object) and can be downloaded separately with streaming support.

## Quality of Life

- All scrape requests and metadata are automatically tracked on a [Web Dashboard ](https://scrapfly.io/docs/monitoring)
- Multi project/scraper support through [Project Management ](https://scrapfly.io/docs/project)
- Experiment with the [Visual API playground ](https://scrapfly.io/dashboard/playground/web-scraper)
- [Status page](https://scrapfly.statuspage.io/) with notification subscription.
- Full API transparency through useful meta headers:
    - **X-Scrapfly-Api-Cost** API Cost billed
    - **X-Scrapfly-Remaining-Api-Credit** Remaining Api Credit, if 0, billed in extra credit
    - **X-Scrapfly-Account-Concurrent-Usage** You current concurrency usage of your account
    - **X-Scrapfly-Account-Remaining-Concurrent-Usage** Maximum concurrency allowed by the account
    - **X-Scrapfly-Project-Concurrent-Usage** Concurrency usage of the project
    - **X-Scrapfly-Project-Remaining-Concurrent-Usage** If the concurrency limit is set on the project otherwise equal to the account concurrency

     Concurrency is defined by your subscription

## Billing

 Scrapfly uses a credit system to bill scrape API requests where each scrape request has a variable cost based on:

- Enabled scrape features and options (browser rendering, blocking bypass etc.).
- Response body type (binary vs text results).
- The [Unblocker](https://scrapfly.io/docs/scrape-api/unblocker) feature can override scrape config details to bypass blocking which can alter the overall cost.

 For more information see [scrape API billing breakdown](https://scrapfly.io/docs/scrape-api/billing).

 Billing is reported in every scrape response and the monitoring dashboard and can be controlled through Scrapfly budget settings. For more see [Web Scraper Billing](https://scrapfly.io/docs/scrape-api/billing).

## Handle Large Object

 Large object `CLOB` for text and `BLOB` are offloaded from the API response to prevent any CPU/RAM issue with your JSON/MSGPACK decoder and increase the efficiency of your scrapers.

 Instead of the actual content in `response.result.content`, you get an URL to download the large object. The URL is valid until the log expire.

- `response.result.format` indicate whether it's a large object by checking if it's `blob` or `clob`
- `response.result.content` contains the url to download the content. This url need to be authenticated with your API Key (Must be the API key that belong to project/env)
- `BLOB` is not base64 encoded like `binary` format, you directly retrieve the binary data and the `Content-Type` header announce the actual type

 A response larger than **256 MB** is not downloaded: the scrape fails with `ERR::SCRAPE::UPSTREAM_WEBSITE_ERROR` and is not retried. It is billed as a scrape, plus the bandwidth it used past the first 1MB, see [billing](https://scrapfly.io/docs/scrape-api/billing#download_limit). With browser rendering, the limit holds for the files the browser downloads, not for the page it renders.

## Errors

 Scrapfly uses conventional HTTP response codes to indicate the success or failure of an API request.

 **Codes in the 2xx** range indicate success.

 **Codes in the 4xx** range indicate an error that failed given the information provided (e.g., a required parameter was omitted, not permitted, max concurrency reached, etc.).

 **Codes in the 5xx** range indicate an error with Scrapfly's servers.

---

 **HTTP 422 - Request Failed** provide extra headers in order to help as much as possible:

- **X-Scrapfly-Reject-Code:** Error Code
- **X-Scrapfly-Reject-Description:** URL to the related documentation
- **X-Scrapfly-Reject-Retryable:** Indicate if the scrape is retryable

> It is important to properly handle HTTP client errors in order to access the error headers and body. These details contain valuable information for troubleshooting, resolving the issue or reaching the support.

###  HTTP Status Code Summary

| 200 - OK | Everything worked as expected. |
|---|---|
| 400 - Bad Request | The request was unacceptable, often due to missing a required parameter or a bad value or a bad format. |
| 401 - Unauthorized | No valid API key provided. |
| 402 - Payment Required | A payment issue occur and need to be resolved |
| 403 - Forbidden | The API key doesn't have permissions to perform the request. |
| 422 - Request Failed | The parameters were valid but the request failed. |
| 429 - Too Many Requests | All free quota used or max allowed concurrency or domain throttled |
| 500, 502, 503 - Server Errors | Something went wrong on Scrapfly's end. |
| 504 - Timeout | The scrape have timeout |
| You can check out the [full error list](https://scrapfly.io/docs/scrape-api/errors) to learn more. |

---

## Specification

 Scrapfly has loads of features and the best way to discover them is through the specification docs below.

 If you have any questions you can check out the [Frequently asked question section](https://scrapfly.io/docs/scrape-api/faq) or see the [support chat](https://scrapfly.io/docs/support).

> **By default, the API has a read timeout of 155 seconds.** To avoid read timeout errors, you must configure your HTTP client to set the read timeout to 155 seconds. If you need a different timeout value, please refer to the documentation for information on [how to control the timeout.](https://scrapfly.io/docs/scrape-api/understand-timeout)

Try out the API directly in your terminal using `curl`:

- [GET](#GET)
- [POST](#POST)
- [PUT](#PUT)
- [PATCH](#PATCH)
- [HEAD](#HEAD)
- [OPTIONS](#OPTIONS)

```
curl -X GET https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key=
```

```
curl -X POST https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key= -H content-type: text/json --data-raw "{\"test\": \"example\"}"
```

```
curl -X PUT https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key= -H content-type: text/json --data-raw "{\"test\": \"example\"}"
```

```
curl -X PATCH https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key= -H content-type: text/json --data-raw "{\"test\": \"example\"}"
```

```
curl -X OPTIONS https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything&country=us&render_js=true&key= -H content-type: text/json --data-raw "{\"test\": \"example\"}"
```

```
curl -I https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key=
```

 Want to try out the API without coding? Check out our visual API player and test/generate code to use our API.

 [Checkout The Web Player](https://scrapfly.io/dashboard/playground/web-scraper)

 The default response format is JSON, and the scraped content is available in `result.content`. Your scrape configuration is present in `config`, and other activated feature information is available in `context`. To get the HTML page directly, refer to the [ `proxified_response` ](https://scrapfly.io/docs/scrape-api/getting-started#api_param_proxified_response) parameter.

 Required Parameters

 [`url`](#api_param_url)

  required

Target URL to scrape. [URL encode ](https://scrapfly.io/web-scraping-tools/urlencode) the value once, as part of the query string

`https://httpbin.dev/anything?q=test`

 [`key`](#api_param_key)

  required

API Key for authentication. Find your key on [dashboard](https://scrapfly.io/docs/project#api-keys)

`scp-live-xxx...`

 Proxy &amp; Location

 [`proxy_pool`](#api_param_proxy_pool)

  popular default: public\_datacenter\_pool

Select proxy pool. See [proxy dashboard](https://scrapfly.io/dashboard/proxy) for available pools and pricing

`public_datacenter_pool` `public_residential_pool`

 [`country`](#api_param_country)

  popular default: random

Proxy country (ISO 3166-1 alpha-2). Supports lists (`us,gb`) and exclusions (`-gb`)

`us` `us,ca,mx` `-gb`

   More details

Without `country`, the proxy is drawn from the pool's countries, the country of the website's domain first when it names one (a `.de` website goes through Germany). The Unblocker can use another country when the website requires it. The country used is in the response, under `context.proxy.country`.

With the [Unblocker](https://scrapfly.io/docs/scrape-api/unblocker#usage), a country you set is kept, unless the website's protection denies access to visitors from that country: the request is then served from another country instead of failing. The country that served the request is `context.proxy.country`, to compare with what you asked for in `config.country`.

**Country selection modes:**

- **Single country:** `country=us`
- **Multiple countries:** `country=us,ca,mx` (one of them is drawn, the country of the website's domain first when it is in the list)
- **Exclusions:** `country=-gb` (exclude UK)

 Uses ISO 3166-1 alpha-2 country codes. See [available countries by proxy pool](https://scrapfly.io/docs/scrape-api/proxy#geo).

 [`lang`](#api_param_lang)

  popular default: proxy location

Page language (sets `Accept-Language` header). Defaults to proxy location language

`en` `fr-FR,en`

   More details

**How it works:**

- Sets the `Accept-Language` HTTP header automatically
- Sending `lang` together with an `Accept-Language` header is refused (400): use `lang`
- Multiple languages can be listed in priority order, separated by commas
- Each value is a language tag (`en`, `fr-FR`, `zh-Hans-CN`), without `;q=` weights

**Examples:**

- `lang=en` - English content
- `lang=fr-FR,en` - French (France) preferred, English fallback
- `lang=en-IN,en-US` - English (India) preferred, US English fallback

 When supported by the target website, the returned content will be in the specified language.

 [`os`](#api_param_os)

 default: null

Operating System. Cannot be set with custom `User-Agent` header

`win11` `mac` `linux` `android` `iphone` `ipad`

   More details

**Desktop:** `win11`, `mac`, `linux`, `chromeos`

**Mobile:** `android`, `iphone`, `ipad`

A mobile value makes the browser a real phone or tablet as a whole: screen, device pixel ratio, GPU, core count and memory all belong to the same device.

To scrape the mobile version of a website, set a mobile value and no custom `User-Agent`: with the Unblocker, a custom Chrome `User-Agent` of a phone or a tablet gives a desktop browser.

**Browser brand support:**

- `chrome` - every platform
- `edge` - `win11` and `android`
- `brave` and `opera` - desktop and `android`

 Brave and Opera on iOS are WebKit wrappers with their own user-agent shape; pairing them with `iphone`/`ipad` is rejected rather than emitting an incoherent fingerprint.

 [`browser_brand`](#api_param_browser_brand)

 default: chrome

Chromium-based browser brand used for fingerprint generation (applies when `render_js=true`). Invalid values are silently dropped and the default applies.

`chrome` `edge` `brave` `opera`

 Request Configuration

 [`headers`](#api_param_headers)

  popular default: \[\]

Custom HTTP headers. [URL encode](https://scrapfly.io/web-scraping-tools/urlencode) the value once, as part of the query string

`headers[Cookie]=test%3D1`

   More details

**Syntax:** `headers[header-name]=value`, the value URL encoded once in the query string

**Examples:**

- `headers[Cookie]=test%3D1%3Bauth%3D1` - Set cookies
- `headers[content-type]=application%2Fjson` - Set content type
- `headers[Authorization]=Bearer%20token123` - Auth header

**Multiple headers:** Pass multiple `headers[]` parameters in the same request.

 Some headers like `Accept-Language` are managed by other parameters (use `lang` instead). Values are URL encoded once: an HTTP client that encodes query parameters for you takes the plain header value, and a value encoded beforehand is encoded twice (a `Cookie` or `Authorization` header then reaches the website with its percent signs).

 With `render_js`, and when the Unblocker loads the page in a headless browser, only `Cookie` and `Referer` are guaranteed to be sent: other custom headers are not guaranteed.

 [`timeout`](#api_param_timeout)

 default: 150000

Timeout in milliseconds, `150000` (150 seconds) at most. A custom value needs `retry=false`: with `retry=true` (the default) the timeout is fixed at `150000` and any other value is refused. See the [timeout documentation](https://scrapfly.io/docs/scrape-api/understand-timeout)

`30000` `120000`

   More details

**Limits with `retry=false`:**

- `unblocker=true` - from `30000` to `150000` (default `75000`)
- `render_js=true` with a `js_scenario` - from `30000` to `150000` (default `60000`)
- `render_js=true` - from `30000` to `150000` (default `90000`)
- No browser, no Unblocker - from `15000` to `150000` (default `150000`)

 A value above `150000`, below the minimum listed above, or sent with `retry=true` is refused with a `400` `ERR::SCRAPE::CONFIG_ERROR`. Set your HTTP client read timeout to the Scrapfly timeout plus 5 seconds.

 [`retry`](#api_param_retry)

 default: true

Retry on failure (network errors, HTTP 5xx). With `retry=true` the `timeout` is fixed at `150000`: set `retry=false` to choose your own

`true` `false`

 Response Format

 [`format`](#api_param_format)

 default: raw

Output format: `raw`, `clean_html`, `json`, `markdown`, `text`. Markdown/text support `:no_links,no_images,only_content` options

`raw` `markdown` `text`

   More details

**Available formats:**

- `raw` - Original HTML as-is
- `clean_html` - Cleaned and sanitized HTML
- `json` - Attempt to parse as JSON
- `markdown` - Convert to Markdown
- `text` - Extract plain text

**Format options (markdown/text only):**

- `format=markdown:no_links` - Remove links
- `format=text:no_images` - Remove image references
- `format=markdown:only_content` - Extract main content only

Combine options: `format=markdown:no_links,no_images,only_content`

 [`proxified_response`](#api_param_proxified_response)

  popular default: false

Return scraped content directly as response body (instead of JSON wrapper). Large objects (CLOB/BLOB) are auto-streamed

`true` `false`

   More details

**When enabled:**

- Page content becomes the response body directly
- Actual HTTP status codes and headers from target are returned
- Works with custom `format` options (JSON, markdown, etc.)
- Large objects (CLOB/BLOB) are streamed automatically

**Available Scrapfly headers:**

- `X-Scrapfly-Content-Format` - Data type (text or binary)
- `X-Scrapfly-Log` - Log ID for debugging
- `X-Scrapfly-Api-Cost` - Credits charged
- `X-Scrapfly-Remaining-Api-Credit` - Remaining credits
- `X-Scrapfly-Reject-Code` - Error code (on failures)

 When using data extraction, extracted data is available in `result.extracted_data` with corresponding content-type.

 Debugging &amp; Tracking

 [`debug`](#api_param_debug)

 default: false

Store API result and take screenshot (if render_js enabled). Enable when contacting support

`true` `false`

 [`correlation_id`](#api_param_correlation_id)

 default: null

Helper ID for grouping scrapes. Can be [filtered in dashboard](https://scrapfly.io/docs/monitoring#filters)

`e3ba784cde0d`

 [`tags`](#api_param_tags)

 default: \[\]

Tags for grouping scrapes in [monitoring dashboard](https://scrapfly.io/docs/monitoring#filters)

`tags[]=jewelery`

 [`dns`](#api_param_dns)

 default: false

Query and retrieve target DNS information

`true` `false`

 [`ssl`](#api_param_ssl)

 default: false

Pull remote SSL certificate and TLS info. Only for `https://` targets

`true` `false`

 [`webhook_name`](#api_param_webhook_name)

  popular default: null

Queue request and redirect response to webhook. Create webhooks in [dashboard](https://scrapfly.io/dashboard/webhook)

`my-webhook-name`

 Data Extraction

 [`extraction_template`](#api_param_extraction_template)

 default: null

Define extraction template for structured data. Use ephemeral (on-the-fly) or stored template by name

`ephemeral:base64(json_template)`

 [`extraction_prompt`](#api_param_extraction_prompt)

  popular default: null

LLM instruction to extract data or ask questions. [URL encode ](https://scrapfly.io/web-scraping-tools/urlencode) the value once, as part of the query string

`Summarize this document`

 [`extraction_model`](#api_param_extraction_model)

  popular default: null

AI auto-extraction for structured data using predefined models

`product` `article` `review_list`

 Unblocker

 [`unblocker`](#api_param_unblocker)

  popular default: false

Enable [Unblocker](https://scrapfly.io/docs/scrape-api/unblocker) (formerly ASP) to bypass blocks and protection systems

`true` `false`

   More details

**Unblocker** automatically handles:

- CAPTCHA challenges and bot detection
- JavaScript challenges (Cloudflare, PerimeterX, etc.)
- Browser fingerprinting and TLS fingerprints
- Rate limiting and access restrictions

 Unblocker dynamically upgrades parameters (proxy_pool, browser) to bypass protection, which can increase API cost. Use `cost_budget` to limit spending.

 [`asp`](#api_param_asp)

  legacy default: false

Previous name of [unblocker](#api_param_unblocker). Still accepted, nothing to migrate. When both are sent, `asp` wins

`true` `false`

 [`cost_budget`](#api_param_cost_budget)

 default: null

Maximum cost of one scrape, in API credits. Read only with [unblocker](#api_param_unblocker) enabled (or a `solve_captcha` step in a `js_scenario`), ignored otherwise. Set the minimum needed to pass your target

`25` `55`

   More details

With `unblocker=true` the Unblocker may retry with other configurations (a residential proxy, browser rendering), which increases the cost. The budget is checked against the cost known before the page is fetched: the proxy network, the request body bandwidth, JavaScript rendering, the Unblocker fee, extraction, the fee of a protected target and captcha solving (a `solve_captcha` step counts as 20 credits until it has run). It is checked before the scrape starts and again before each upgrade, and an upgrade the budget cannot pay is not made.

- The bandwidth of the response is known only once the page is downloaded, so no budget limits it
- Without `unblocker=true` and without a `solve_captcha` step the parameter is accepted and ignored: a request with `render_js=true` and `cost_budget=1` is not refused and bills its 6 credits
- A scrape the budget cannot pay fails with [`ERR::SCRAPE::COST_BUDGET_LIMIT`](https://scrapfly.io/docs/scrape-api/error/ERR::SCRAPE::COST_BUDGET_LIMIT) (HTTP 422, not retryable). Nothing is billed when it is refused before a request reached the target
- It is a Web Scraping API parameter: the Screenshot API ignores it

 Set the minimum budget your target needs. The credits in the refusal message are the forecast at that moment and can be lower than what the scrape bills (a refusal before a proxy exit is drawn leaves the credit of the proxy out), so raise the budget until the request passes.

 Headless Browser / Javascript Rendering

 [`render_js`](#api_param_render_js)

  popular default: false

Enable browser rendering to execute JavaScript and render dynamic content

`true` `false`

 [`rendering_wait`](#api_param_rendering_wait)

 default: 1000

Delay in milliseconds after page load. Only for HTML pages

`1000` `5000`

 [`wait_for_selector`](#api_param_wait_for_selector)

  popular default: null

Wait until CSS/XPath selector or XHR pattern visible. Use `xhr:` prefix for XHR patterns

`body` `#content` `//button` `xhr:/api/*`

   More details

**Supported selector types:**

- **CSS Selector** - Standard CSS selectors like `body`, `input[type="submit"]`
- **XPath Selector** - XPath expressions like `//button[contains(text(),"Go")]`
- **XHR Pattern** - Network request patterns prefixed with `xhr:`

**XHR Pattern matching:**

- Prefix matching: `xhr:/page/reviews`
- Wildcard matching: `xhr:/page/*`

 Only executed on `HTML` pages. The selector is waited for 15 seconds: when it does not show up in that time, the scrape ends with [`ERR::SCRAPE::DOM_SELECTOR_NOT_FOUND`](https://scrapfly.io/docs/scrape-api/error/ERR::SCRAPE::DOM_SELECTOR_NOT_FOUND) (HTTP 422) and the page as it was rendered is returned in `result.content`. The scrape ran, so it is billed. When the `timeout` of the scrape leaves less than that, the scrape ends with [`ERR::SCRAPE::OPERATION_TIMEOUT`](https://scrapfly.io/docs/scrape-api/error/ERR::SCRAPE::OPERATION_TIMEOUT) instead.

 [`js`](#api_param_js)

 default: null

JavaScript to execute (base64 encoded, max 16KB). [Encode here ](https://scrapfly.io/web-scraping-tools/base64)

`cmV0dXJuIG5hdmlnYXRvci51c2VyQWdlbnQ`

   More details

**Execution behavior:**

- If `wait_for_selector` is defined, the script executes **after** the selector is found
- Use JavaScript `await` to prevent early return when waiting for data
- Return values are available in the API response

 Only executed on `HTML` pages. Maximum script size is 16KB before base64 encoding.

 [`screenshots`](#api_param_screenshots)

  popular default: \[\]

Capture screenshots of fullpage or specific elements. Key=name, value=selector or `fullpage`

`screenshots[page]=fullpage` `screenshots[price]=#price`

   More details

**Capture options:**

- `fullpage` - Captures the entire page including scrolled content
- **CSS selector** - Captures only the matching element (e.g., `#price`)
- **XPath selector** - Captures element by XPath expression

**Multiple screenshots:** You can take multiple screenshots of different areas by specifying different names:

- `screenshots[page]=fullpage`
- `screenshots[price]=#product-price`
- `screenshots[reviews]=.reviews-section`

 [`screenshot_flags`](#api_param_screenshot_flags)

 default: \[\]

Screenshot options: `load_images`, `dark_mode`, `block_banners`, `high_quality`, `print_media_format`

`load_images` `block_banners,high_quality`

   More details

**Available flags:**

- `load_images` - Load images (extra bandwidth cost applies)
- `dark_mode` - Enable dark mode display
- `block_banners` - Block cookie banners and overlays
- `high_quality` - No compression on output image
- `print_media_format` - Render page in print mode

Combine multiple flags with commas: `screenshot_flags=load_images,block_banners,high_quality`

 [`js_scenario`](#api_param_js_scenario)

 default: null

JSON scenario to navigate or perform page actions. [Must be base64 encoded ](https://scrapfly.io/web-scraping-tools/base64)

`eydjbGljayc6IHsnc2VsZWN0b3InOiAnI3N1Ym1pdCd9fQ`

   More details

**Available scenario actions:**

- `click` - Click on elements
- `fill` - Fill input fields with text
- `wait` - Wait for specified milliseconds
- `scroll` - Scroll the page or element
- `execute` - Execute custom JavaScript
- `wait_for_selector` - Wait for element to appear
- `wait_for_navigation` - Wait for page navigation
- `solve_captcha` - Solve the captcha on the page

**Example scenario (before base64):**

 `[{"click": {"selector": "#login-btn"}}, {"fill": {"selector": "#username", "value": "test"}}]`

 [`geolocation`](#api_param_geolocation)

 default: null

Spoof browser geolocation. Format: `latitude,longitude`

`48.856614,2.3522219` `40.712784,-74.005941`

 [`auto_scroll`](#api_param_auto_scroll)

 default: false

Auto-scroll to bottom to trigger lazy-loaded content

`true` `false`

 [`rendering_stage`](#api_param_rendering_stage)

 default: complete

Page load stage to wait for. Use `domcontentloaded` for faster scrapes

`complete` `domcontentloaded`

 Caching Options

 [`cache`](#api_param_cache)

  popular default: false

Enable caching. Returns cached content if HIT, otherwise scrapes and caches

`true` `false`

 [`cache_ttl`](#api_param_cache_ttl)

 default: 86400

Cache time-to-live in seconds. Expired cache triggers fresh scrape

`60` `3600` `86400`

 [`cache_clear`](#api_param_cache_clear)

 default: false

Force cache refresh on this request

`true` `false`

 Session Management

 [`session`](#api_param_session)

  popular default: null

Session name to persist cookies, fingerprint, and proxy across scrapes. Letters, digits, hyphen and underscore, max 255 chars (a Cloud Browser session is `cb:` and its id)

`my-session-123`

   More details

**Session automatically persists:**

- **Cookies** - Login sessions, preferences, cart data
- **Browser fingerprint** - Consistent identity across requests
- **Proxy IP** - Same IP when possible (see `session_sticky_proxy`)

**Use cases:**

- Multi-step authentication flows
- Shopping cart persistence
- Pagination with session-based state

 A session name holds letters, digits, hyphens (`-`) and underscores (`_`), starts with a letter or a digit, and is 255 characters at most. Any other character, a dot or a space for example, is refused with a `400` `ERR::SCRAPE::CONFIG_ERROR`, except the `cb:` prefix of a [Cloud Browser session](https://scrapfly.io/docs/scrape-api/session#cloud-browser-session). Sessions are automatically cleaned after inactivity.

 [`session_sticky_proxy`](#api_param_session_sticky_proxy)

 default: true

Best effort to reuse same proxy IP within session

`true` `false`

   More details

When enabled, the system attempts to use the same proxy IP address for all requests within a session. This is useful for:

- Websites that track IP consistency
- Rate-limited sites that count per-IP
- Session-based authentication tied to IP

 This is a best effort feature. The same IP is not guaranteed if the proxy becomes unavailable.

 Setting this to `false` while `unblocker` is enabled is rejected with HTTP 400. The anti-bot clearance is tied to the IP that passed the challenge, so the IP cannot change within an unblocker session.
