# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Bring Your Own Proxy](https://scrapfly.io/docs/cloud-browser-api/bring-your-own-proxy) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Getting Started with Scrapfly [ View as markdown ](https://scrapfly.io/?view=markdown) Copy for LLM Copy for LLM [ Open in ChatGPT ](https://chatgpt.com/?hints=search&prompt=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fscrape-api%2Fgetting-started%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Claude ](https://claude.ai/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fscrape-api%2Fgetting-started%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Perplexity ](https://www.perplexity.ai/search/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fscrape-api%2Fgetting-started%20so%20I%20can%20ask%20questions%20about%20it.) Discover how to use Scrapfly API - the basics, available parameters and features, error handling and other information related to the API use. Minimal API call is a GET, POST, PUT, PATCH or HEAD request with `url` and `key` parameters: ``` https://api.scrapfly.io/scrape?url=&key= ``` [ See Usage ](#spec) ## On Steroids - Smart defaults - **scrape without being blocked** . Scrapfly pre-configures user-agent and other request headers. - [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) feature bypasses all anti-scraping systems. - By default, the API responds in JSON. Though, a more efficient `msgpack` format is also available by setting the accept: application/msgpack header. - Text content is returned as utf-8 while binary is encoded in base64, so you can scrape any kind of data (pdf, zip, etc) - Gzip compression is available through content-encoding: gzip header. - Ability to [debug and replay scrape requests](https://scrapfly.io/docs/scrape-api/debug) from the dashboard log page and API. - [Handle large payload](#handle_large_object), large text response greater than 5MB are called "CLOB" (Character Large Object) and binary are called "BLOB" (Binary Large Object) and can be downloaded separately with streaming support. ## Quality of Life - All scrape requests and metadata are automatically tracked on a [Web Dashboard ](https://scrapfly.io/docs/monitoring) - Multi project/scraper support through [Project Management ](https://scrapfly.io/docs/project) - Experiment with the [Visual API playground ](https://scrapfly.io/dashboard/playground/web-scraper) - [Status page](https://scrapfly.statuspage.io/) with notification subscription. - Full API transparency through useful meta headers: - **X-Scrapfly-Api-Cost** API Cost billed - **X-Scrapfly-Remaining-Api-Credit** Remaining Api Credit, if 0, billed in extra credit - **X-Scrapfly-Account-Concurrent-Usage** You current concurrency usage of your account - **X-Scrapfly-Account-Remaining-Concurrent-Usage** Maximum concurrency allowed by the account - **X-Scrapfly-Project-Concurrent-Usage** Concurrency usage of the project - **X-Scrapfly-Project-Remaining-Concurrent-Usage** If the concurrency limit is set on the project otherwise equal to the account concurrency Concurrency is defined by your subscription ## Billing Scrapfly uses a credit system to bill scrape API requests where each scrape request has a variable cost based on: - Enabled scrape features and options (browser rendering, blocking bypass etc.). - Response body type (binary vs text results). - [ASP](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) feature can override scrape config details to bypass blocking which can alter the overall cost. For more information see [scrape API billing breakdown](https://scrapfly.io/docs/scrape-api/billing#billing). Billing is reported in every scrape response and the monitoring dashboard and can be controlled through Scrapfly budget settings. For more see [Web Scraper Billing](https://scrapfly.io/docs/scrape-api/billing). ## Handle Large Object Large object `CLOB` for text and `BLOB` are offloaded from the API response to prevent any CPU/RAM issue with your JSON/MSGPACK decoder and increase the efficiency of your scrapers. Instead of the actual content in `response.result.content`, you get an URL to download the large object. The URL is valid until the log expire. - `response.result.format` indicate whether it's a large object by checking if it's `blob` or `clob` - `response.result.content` contains the url to download the content. This url need to be authenticated with your API Key (Must be the API key that belong to project/env) - `BLOB` is not base64 encoded like `binary` format, you directly retrieve the binary data and the `Content-Type` header announce the actual type ## Errors Scrapfly uses conventional HTTP response codes to indicate the success or failure of an API request. **Codes in the 2xx** range indicate success. **Codes in the 4xx** range indicate an error that failed given the information provided (e.g., a required parameter was omitted, not permitted, max concurrency reached, etc.). **Codes in the 5xx** range indicate an error with Scrapfly's servers. --- **HTTP 422 - Request Failed** provide extra headers in order to help as much as possible: - **X-Scrapfly-Reject-Code:** Error Code - **X-Scrapfly-Reject-Description:** URL to the related documentation - **X-Scrapfly-Reject-Retryable:** Indicate if the scrape is retryable > It is important to properly handle HTTP client errors in order to access the error headers and body. These details contain valuable information for troubleshooting, resolving the issue or reaching the support. ### HTTP Status Code Summary | 200 - OK | Everything worked as expected. | |---|---| | 400 - Bad Request | The request was unacceptable, often due to missing a required parameter or a bad value or a bad format. | | 401 - Unauthorized | No valid API key provided. | | 402 - Payment Required | A payment issue occur and need to be resolved | | 403 - Forbidden | The API key doesn't have permissions to perform the request. | | 422 - Request Failed | The parameters were valid but the request failed. | | 429 - Too Many Requests | All free quota used or max allowed concurrency or domain throttled | | 500, 502, 503 - Server Errors | Something went wrong on Scrapfly's end. | | 504 - Timeout | The scrape have timeout | | You can check out the [full error list](https://scrapfly.io/docs/scrape-api/errors) to learn more. | --- ## Specification Scrapfly has loads of features and the best way to discover them is through the specification docs below. If you have any questions you can check out the [Frequently asked question section](https://scrapfly.io/docs/scrape-api/faq) or see the [support chat](https://scrapfly.io/docs/support). > **By default, the API has a read timeout of 155 seconds.** To avoid read timeout errors, you must configure your HTTP client to set the read timeout to 155 seconds. If you need a different timeout value, please refer to the documentation for information on [how to control the timeout.](https://scrapfly.io/docs/scrape-api/understand-timeout) Try out the API directly in your terminal using `curl`: - [GET](#GET) - [POST](#POST) - [PUT](#PUT) - [PATCH](#PATCH) - [HEAD](#HEAD) - [OPTIONS](#OPTIONS) ``` curl -X GET https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key= ``` ``` curl -X POST https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key= -H content-type: text/json --data-raw "{\"test\": \"example\"}" ``` ``` curl -X PUT https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key= -H content-type: text/json --data-raw "{\"test\": \"example\"}" ``` ``` curl -X PATCH https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key= -H content-type: text/json --data-raw "{\"test\": \"example\"}" ``` ``` curl -X OPTIONS https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything&country=us&render_js=true&key= -H content-type: text/json --data-raw "{\"test\": \"example\"}" ``` ``` curl -I https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key= ``` Want to try out the API without coding? Check out our visual API player and test/generate code to use our API. [Checkout The Web Player](https://scrapfly.io/dashboard/playground/web-scraper) The default response format is JSON, and the scraped content is available in `result.content`. Your scrape configuration is present in `config`, and other activated feature information is available in `context`. To get the HTML page directly, refer to the [ `proxified_response` ](https://scrapfly.io/docs/scrape-api/getting-started#api_param_proxified_response) parameter. Required Parameters [`url`](#api_param_url) required Target URL to scrape. [Must be URL encoded ](https://scrapfly.io/web-scraping-tools/urlencode) `https://httpbin.dev/anything?q=test` [`key`](#api_param_key) required API Key for authentication. Find your key on [dashboard](https://scrapfly.io/docs/project#api-keys) `scp-live-xxx...` Proxy & Location [`proxy_pool`](#api_param_proxy_pool) popular default: public\_datacenter\_pool [ ](https://scrapfly.io/docs/scrape-api/proxy?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/proxy "Docs") Select proxy pool. See [proxy dashboard](https://scrapfly.io/dashboard/proxy) for available pools and pricing `public_datacenter_pool` `public_residential_pool` [`country`](#api_param_country) popular default: random [ ](https://scrapfly.io/docs/scrape-api/proxy?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/proxy "Docs") Proxy country (ISO 3166-1 alpha-2). Supports exclusions (`-gb`) and weighted distribution (`us:10,gb:5`) `us` `us,ca,mx` `-gb` More details**Country selection modes:** - **Single country:** `country=us` - **Multiple countries:** `country=us,ca,mx` (random selection) - **Exclusions:** `country=-gb` (exclude UK) - **Weighted:** `country=us:10,gb:5` (2x more US than UK) Uses ISO 3166-1 alpha-2 country codes. See [available countries by proxy pool](https://scrapfly.io/docs/scrape-api/proxy#geo). [`lang`](#api_param_lang) popular default: proxy location [ ](https://scrapfly.io/docs/scrape-api/custom?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/custom "Docs") Page language (sets `Accept-Language` header). Defaults to proxy location language `en` `fr-FR,en` More details**How it works:** - Sets the `Accept-Language` HTTP header automatically - The `Accept-Language` header cannot be set manually via `headers` parameter - Multiple languages can be listed in priority order **Examples:** - `lang=en` - English content - `lang=fr-FR,en` - French (France) preferred, English fallback - `lang=en-IN,en-US` - English (India) preferred, US English fallback When supported by the target website, the returned content will be in the specified language. [`os`](#api_param_os) default: null [ ](https://scrapfly.io/docs/scrape-api/custom?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/custom "Docs") Operating System. Cannot be set with custom `User-Agent` header `win11` `mac` `linux` Request Configuration [`headers`](#api_param_headers) popular default: \[\] [ ](https://scrapfly.io/docs/scrape-api/custom?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/custom "Docs") Custom HTTP headers. [Must be URL encoded](https://scrapfly.io/web-scraping-tools/urlencode) `headers[Cookie]=test%3D1` More details**Syntax:** `headers[header-name]=encoded-value` **Examples:** - `headers[Cookie]=test%3D1%3Bauth%3D1` - Set cookies - `headers[content-type]=application%2Fjson` - Set content type - `headers[Authorization]=Bearer%20token123` - Auth header **Multiple headers:** Pass multiple `headers[]` parameters in the same request. Some headers like `Accept-Language` are managed by other parameters (use `lang` instead). Values must be URL encoded. [`timeout`](#api_param_timeout) default: 150000 [ ](https://scrapfly.io/docs/scrape-api/understand-timeout?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/understand-timeout "Docs") Timeout in milliseconds. See [timeout documentation](https://scrapfly.io/docs/scrape-api/understand-timeout) `30000` `120000` [`retry`](#api_param_retry) default: true Retry on failure (network errors, HTTP 5xx). Has impact on timeout `true` `false` Response Format [`format`](#api_param_format) default: raw Output format: `raw`, `clean_html`, `json`, `markdown`, `text`. Markdown/text support `:no_links,no_images,only_content` options `raw` `markdown` `text` More details**Available formats:** - `raw` - Original HTML as-is - `clean_html` - Cleaned and sanitized HTML - `json` - Attempt to parse as JSON - `markdown` - Convert to Markdown - `text` - Extract plain text **Format options (markdown/text only):** - `format=markdown:no_links` - Remove links - `format=text:no_images` - Remove image references - `format=markdown:only_content` - Extract main content only Combine options: `format=markdown:no_links,no_images,only_content` [`proxified_response`](#api_param_proxified_response) popular default: false Return scraped content directly as response body (instead of JSON wrapper). Large objects (CLOB/BLOB) are auto-streamed `true` `false` More details**When enabled:** - Page content becomes the response body directly - Actual HTTP status codes and headers from target are returned - Works with custom `format` options (JSON, markdown, etc.) - Large objects (CLOB/BLOB) are streamed automatically **Available Scrapfly headers:** - `X-Scrapfly-Content-Format` - Data type (text or binary) - `X-Scrapfly-Log` - Log ID for debugging - `X-Scrapfly-Api-Cost` - Credits charged - `X-Scrapfly-Remaining-Api-Credit` - Remaining credits - `X-Scrapfly-Reject-Code` - Error code (on failures) When using data extraction, extracted data is available in `result.extracted_data` with corresponding content-type. Debugging & Tracking [`debug`](#api_param_debug) default: false [ ](https://scrapfly.io/docs/scrape-api/debug?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/debug "Docs") Store API result and take screenshot (if render_js enabled). Enable when contacting support `true` `false` [`correlation_id`](#api_param_correlation_id) default: null Helper ID for grouping scrapes. Can be [filtered in dashboard](https://scrapfly.io/docs/monitoring#filters) `e3ba784cde0d` [`tags`](#api_param_tags) default: \[\] Tags for grouping scrapes in [monitoring dashboard](https://scrapfly.io/docs/monitoring#filters) `tags[]=jewelery` [`dns`](#api_param_dns) default: false [ ](https://scrapfly.io/docs/scrape-api/dns?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/dns "Docs") Query and retrieve target DNS information `true` `false` [`ssl`](#api_param_ssl) default: false [ ](https://scrapfly.io/docs/scrape-api/ssl?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/ssl "Docs") Pull remote SSL certificate and TLS info. Only for `https://` targets `true` `false` [`webhook_name`](#api_param_webhook_name) popular default: null [ ](https://scrapfly.io/docs/scrape-api/webhook?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/webhook "Docs") Queue request and redirect response to webhook. Create webhooks in [dashboard](https://scrapfly.io/dashboard/webhook) `my-webhook-name` Data Extraction [`extraction_template`](#api_param_extraction_template) default: null [ ](https://scrapfly.io/docs/extraction-api/rules-and-template?iframe=1 "Preview") [ ](https://scrapfly.io/docs/extraction-api/rules-and-template "Docs") Define extraction template for structured data. Use ephemeral (on-the-fly) or stored template by name `ephemeral:base64(json_template)` [`extraction_prompt`](#api_param_extraction_prompt) popular default: null [ ](https://scrapfly.io/docs/extraction-api/llm-prompt?iframe=1 "Preview") [ ](https://scrapfly.io/docs/extraction-api/llm-prompt "Docs") LLM instruction to extract data or ask questions. [Must be URL encoded ](https://scrapfly.io/web-scraping-tools/urlencode) `Summarize this document` [`extraction_model`](#api_param_extraction_model) popular default: null [ ](https://scrapfly.io/docs/extraction-api/automatic-ai?iframe=1 "Preview") [ ](https://scrapfly.io/docs/extraction-api/automatic-ai "Docs") AI auto-extraction for structured data using predefined models `product` `article` `review_list` Anti Scraping Protection [`asp`](#api_param_asp) popular default: false [ ](https://scrapfly.io/docs/scrape-api/anti-scraping-protection?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/anti-scraping-protection "Docs") Enable [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) to bypass blocks and protection systems `true` `false` More details**Anti Scraping Protection** automatically handles: - CAPTCHA challenges and bot detection - JavaScript challenges (Cloudflare, PerimeterX, etc.) - Browser fingerprinting and TLS fingerprints - Rate limiting and access restrictions ASP dynamically upgrades parameters (proxy_pool, browser) to bypass protection, which can increase API cost. Use `cost_budget` to limit spending. [`cost_budget`](#api_param_cost_budget) default: null [ ](https://scrapfly.io/docs/scrape-api/anti-scraping-protection?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/anti-scraping-protection "Docs") Limit ASP retry cost. ASP upgrades params dynamically; set budget to control spending. Min value needed to pass target `25` `55` More detailsWhen `asp=true`, the system may retry with different configurations (residential proxies, browser rendering) which increases cost. Set a budget to: - Control maximum spending per request - Fail fast if target requires expensive bypass - Make costs more predictable Set the minimum budget needed for your target. If budget is too low, the request will be rejected without attempting bypass. Headless Browser / Javascript Rendering [`render_js`](#api_param_render_js) popular default: false [ ](https://scrapfly.io/docs/scrape-api/javascript-rendering?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/javascript-rendering "Docs") Enable browser rendering to execute JavaScript and render dynamic content `true` `false` [`rendering_wait`](#api_param_rendering_wait) default: 1000 [ ](https://scrapfly.io/docs/scrape-api/javascript-rendering?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/javascript-rendering "Docs") Delay in milliseconds after page load. Only for HTML pages `1000` `5000` [`wait_for_selector`](#api_param_wait_for_selector) popular default: null [ ](https://scrapfly.io/docs/scrape-api/javascript-rendering?iframe=1#rendering_wait "Preview") [ ](https://scrapfly.io/docs/scrape-api/javascript-rendering#rendering_wait "Docs") Wait until CSS/XPath selector or XHR pattern visible. Use `xhr:` prefix for XHR patterns `body` `#content` `//button` `xhr:/api/*` More details**Supported selector types:** - **CSS Selector** - Standard CSS selectors like `body`, `input[type="submit"]` - **XPath Selector** - XPath expressions like `//button[contains(text(),"Go")]` - **XHR Pattern** - Network request patterns prefixed with `xhr:` **XHR Pattern matching:** - Prefix matching: `xhr:/page/reviews` - Wildcard matching: `xhr:/page/*` Only executed on `HTML` pages. If the selector is not found, the scrape will timeout. [`js`](#api_param_js) default: null [ ](https://scrapfly.io/docs/scrape-api/javascript-rendering?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/javascript-rendering "Docs") JavaScript to execute (base64 encoded, max 16KB). [Encode here ](https://scrapfly.io/web-scraping-tools/base64) `cmV0dXJuIG5hdmlnYXRvci51c2VyQWdlbnQ` More details**Execution behavior:** - If `wait_for_selector` is defined, the script executes **after** the selector is found - Use JavaScript `await` to prevent early return when waiting for data - Return values are available in the API response Only executed on `HTML` pages. Maximum script size is 16KB before base64 encoding. [`screenshots`](#api_param_screenshots) popular default: \[\] [ ](https://scrapfly.io/docs/scrape-api/screenshot?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/screenshot "Docs") Capture screenshots of fullpage or specific elements. Key=name, value=selector or `fullpage` `screenshots[page]=fullpage` `screenshots[price]=#price` More details**Capture options:** - `fullpage` - Captures the entire page including scrolled content - **CSS selector** - Captures only the matching element (e.g., `#price`) - **XPath selector** - Captures element by XPath expression **Multiple screenshots:** You can take multiple screenshots of different areas by specifying different names: - `screenshots[page]=fullpage` - `screenshots[price]=#product-price` - `screenshots[reviews]=.reviews-section` [`screenshot_flags`](#api_param_screenshot_flags) default: \[\] [ ](https://scrapfly.io/docs/scrape-api/screenshot?iframe=1#options "Preview") [ ](https://scrapfly.io/docs/scrape-api/screenshot#options "Docs") Screenshot options: `load_images`, `dark_mode`, `block_banners`, `high_quality`, `print_media_format` `load_images` `block_banners,high_quality` More details**Available flags:** - `load_images` - Load images (extra bandwidth cost applies) - `dark_mode` - Enable dark mode display - `block_banners` - Block cookie banners and overlays - `high_quality` - No compression on output image - `print_media_format` - Render page in print mode Combine multiple flags with commas: `screenshot_flags=load_images,block_banners,high_quality` [`js_scenario`](#api_param_js_scenario) default: null [ ](https://scrapfly.io/docs/scrape-api/javascript-scenario?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/javascript-scenario "Docs") JSON scenario to navigate or perform page actions. [Must be base64 encoded ](https://scrapfly.io/web-scraping-tools/base64) `eydjbGljayc6IHsnc2VsZWN0b3InOiAnI3N1Ym1pdCd9fQ` More details**Available scenario actions:** - `click` - Click on elements - `fill` - Fill input fields with text - `wait` - Wait for specified milliseconds - `scroll` - Scroll the page or element - `execute` - Execute custom JavaScript - `wait_for_selector` - Wait for element to appear - `wait_for_navigation` - Wait for page navigation **Example scenario (before base64):** `[{"click": {"selector": "#login-btn"}}, {"fill": {"selector": "#username", "value": "test"}}]` [`geolocation`](#api_param_geolocation) default: null Spoof browser geolocation. Format: `latitude,longitude` `48.856614,2.3522219` `40.712784,-74.005941` [`auto_scroll`](#api_param_auto_scroll) default: false Auto-scroll to bottom to trigger lazy-loaded content `true` `false` [`rendering_stage`](#api_param_rendering_stage) default: complete Page load stage to wait for. Use `domcontentloaded` for faster scrapes `complete` `domcontentloaded` Caching Options [`cache`](#api_param_cache) popular default: false [ ](https://scrapfly.io/docs/scrape-api/cache?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/cache "Docs") Enable caching. Returns cached content if HIT, otherwise scrapes and caches `true` `false` [`cache_ttl`](#api_param_cache_ttl) default: 86400 [ ](https://scrapfly.io/docs/scrape-api/cache?iframe=1#ttl_eviction "Preview") [ ](https://scrapfly.io/docs/scrape-api/cache#ttl_eviction "Docs") Cache time-to-live in seconds. Expired cache triggers fresh scrape `60` `3600` `86400` [`cache_clear`](#api_param_cache_clear) default: false [ ](https://scrapfly.io/docs/scrape-api/cache?iframe=1#ttl_eviction "Preview") [ ](https://scrapfly.io/docs/scrape-api/cache#ttl_eviction "Docs") Force cache refresh on this request `true` `false` Session Management [`session`](#api_param_session) popular default: null [ ](https://scrapfly.io/docs/scrape-api/session?iframe=1 "Preview") [ ](https://scrapfly.io/docs/scrape-api/session "Docs") Session name to persist cookies, fingerprint, and proxy across scrapes. Alphanumeric, max 255 chars `my-session-123` More details**Session automatically persists:** - **Cookies** - Login sessions, preferences, cart data - **Browser fingerprint** - Consistent identity across requests - **Proxy IP** - Same IP when possible (see `session_sticky_proxy`) **Use cases:** - Multi-step authentication flows - Shopping cart persistence - Pagination with session-based state Session name must be alphanumeric, maximum 255 characters. Sessions are automatically cleaned after inactivity. [`session_sticky_proxy`](#api_param_session_sticky_proxy) default: true [ ](https://scrapfly.io/docs/scrape-api/session?iframe=1#usage "Preview") [ ](https://scrapfly.io/docs/scrape-api/session#usage "Docs") Best effort to reuse same proxy IP within session `true` `false` More detailsWhen enabled, the system attempts to use the same proxy IP address for all requests within a session. This is useful for: - Websites that track IP consistency - Rate-limited sites that count per-IP - Session-based authentication tied to IP This is a best effort feature. The same IP is not guaranteed if the proxy becomes unavailable.