# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Unblocker (formerly ASP)](https://scrapfly.io/docs/scrape-api/unblocker) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Batch (Multi-URL Scraping)](https://scrapfly.io/docs/scrape-api/batch) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Schedule](https://scrapfly.io/docs/scrape-api/schedule) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Search](https://scrapfly.io/docs/crawler-api/search) - [Prompt & Extract](https://scrapfly.io/docs/crawler-api/prompt) - [Auto Refresh](https://scrapfly.io/docs/crawler-api/refresh) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Schedule](https://scrapfly.io/docs/crawler-api/schedule) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Schedule](https://scrapfly.io/docs/screenshot-api/schedule) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Data API #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [Captcha Solver](https://scrapfly.io/docs/cloud-browser-api/captcha-solver) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) - [Credential Vault](https://scrapfly.io/docs/cloud-browser-api/credential-vault) - [Native Browser MCP](https://scrapfly.io/docs/cloud-browser-api/mcp) - [DevTools Protocol](https://scrapfly.io/docs/cloud-browser-api/cdp-reference) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [1Password](https://scrapfly.io/docs/cloud-browser-api/1password) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Proxy Saver Protocols The Proxy Saver supports all standard and modern proxy protocols, ensuring compatibility with any browser or web scraping tool. Below are the details and specificities of each protocol's usage. > While all the provided examples account for the possibility that the Proxy Saver certificates may not be properly installed, the recommended approach is to [configure them correctly](https://scrapfly.io/docs/proxy-saver/certificates) rather than bypassing SSL errors. ## HTTP HTTP is the simplest proxy protocol and is universally compatible with all browsers and libraries. Below are some examples: - [Curl](#basic-use-curl-http) - [Playwright (js)](#scrape-http-proxy-playwright-js) - [Playwright (python)](#scrape-http-proxy-playwright-python) ``` curl \ --insecure \ --proxy http://proxyId-XXX:scp-live-XXX@proxy-saver.scrapfly.io:3333 \ https://httpbin.dev/anything ``` ``` const {chromium} = require("playwright"); (async () => { try { const browser = await chromium.launch({ proxy: { server: 'http://proxy-saver.scrapfly.io:3333', username: 'proxyId-XXX', password: 'scp-live-XXX' } }); const context = await browser.newContext({ignoreHTTPSErrors: true}); const page = await context.newPage(); await page.goto("https://httpbin.dev/anything"); const html = await page.content(); console.log(html); await browser.close(); } catch (error) { console.error("An error occurred:", error); } })(); ``` ``` from playwright.sync_api import sync_playwright def main(): try: with sync_playwright() as p: browser = p.chromium.launch( proxy={ "server": "http://proxy-saver.scrapfly.io:3333", "username": "proxyId-XXX", "password": "scp-live-XXX", } ) context = browser.new_context(ignore_https_errors=True) page = context.new_page() page.goto("https://httpbin.dev/anything") html = page.content() print(html) except Exception as e: print("An error occurred:", e) finally: browser.close() if __name__ == "__main__": main() ``` ## HTTPS / HTTP2 Using HTTP as a proxy protocol is straightforward, but opting for HTTPS ensures that your credentials are securely encrypted from the very start of the CONNECT phase. By default, curl will use HTTP/1.1 over TLS unless you have a recent version that supports HTTP/2 and explicitly specify the necessary flags. Chrome, on the other hand, defaults to HTTP/2 but will gracefully downgrade to HTTP/1.1 over TLS if needed. The Proxy Saver natively supports HTTP/2 while also maintaining compatibility with HTTP/1.1 over TLS to accommodate the broadest range of browser implementations. - [Curl (HTTPS)](#basic-use-curl-https) - [Curl (HTTP2)](#basic-use-curl-http2) - [Playwright (js)](#scrape-https-proxy-playwright-js) - [Playwright (python)](#scrape-https-proxy-playwright-python) ``` curl \ --insecure --proxy-insecure \ --proxy https://proxyId-XXX:scp-live-XXX@proxy-saver.scrapfly.io:3333 \ https://httpbin.dev/anything ``` ``` curl \ --insecure --proxy-insecure \ --proxy-http2 \ --proxy https://proxyId-XXX:scp-live-XXX@proxy-saver.scrapfly.io:3333 \ https://httpbin.dev/anything ``` ``` const {chromium} = require("playwright"); (async () => { try { const browser = await chromium.launch({ proxy: { server: 'https://proxy-saver.scrapfly.io:3333', username: 'proxyId-XXX', password: 'scp-live-XXX' }, args: ['--ignore-certificate-errors'], }); const context = await browser.newContext({ignoreHTTPSErrors: true}); const page = await context.newPage(); await page.goto("https://httpbin.dev/anything"); const html = await page.content(); console.log(html); await browser.close(); } catch (error) { console.error("An error occurred:", error); } })(); ``` ``` from playwright.sync_api import sync_playwright def main(): try: with sync_playwright() as p: browser = p.chromium.launch( proxy={ "server": "https://proxy-saver.scrapfly.io:3333", "username": "proxyId-XXX", "password": "scp-live-XXX", }, args=["--ignore-certificate-errors"] ) context = browser.new_context(ignore_https_errors=True) page = context.new_page() page.goto("https://httpbin.dev/anything") html = page.content() print(html) except Exception as e: print("An error occurred:", e) finally: browser.close() if __name__ == "__main__": main() ``` ## SOCKS5 Socks5/h is another popular protocol used for HTTP proxying ### Socks5 auth mechanism For browsers and libraries that support native SOCKS5 authentication, the Proxy Saver does as well, ensuring seamless integration. - [Curl (SOCKS5 Auth)](#basic-use-curl-https) ``` curl \ --insecure \ --proxy socks5h://proxyId-XXX:scp-live-XXX@proxy-saver.scrapfly.io:3333 \ https://httpbin.dev/anything ``` However, not all browsers support this mechanism. > While Chrome and Chromium do not support the SOCKS5 authentication mechanism (and have no plans to), Scrapfly offers a reliable workaround. For browsers and libraries that do not support SOCKS5 authentication natively, Scrapfly offers a simple workaround using a custom header. Simply include your authentication string in the `X-SAVER-AUTH` header. This custom authentication mechanism enables the use of the SOCKS5 protocol with any library that supports it. Below are some usage examples: - [Curl (X-SAVER-AUTH)](#basic-use-curl-socks5h-x-auth) - [Playwright (js)](#scrape-socks5-proxy-playwright-js) - [Playwright (python)](#scrape-socks5-proxy-playwright-python) ``` curl \ --insecure \ --proxy socks5h://proxy-saver.scrapfly.io:3333 \ --header "X-SAVER-AUTH: proxyId-XXX:scp-live-XXX" \ https://httpbin.dev/anything ``` ``` const { chromium } = require("playwright"); const xSaverAuth = { 'X-SAVER-AUTH': 'proxyId-XXX:scp-live-XXX'}; (async () => { try { const browser = await chromium.launch({ proxy: { server: 'socks5://proxy-saver.scrapfly.io:3333', } }); const context = await browser.newContext({ ignoreHTTPSErrors: true }); await context.setExtraHTTPHeaders(xSaverAuth); const page = await context.newPage(); await page.goto("https://httpbin.dev/anything"); const html = await page.content(); console.log(html); await browser.close(); } catch (error) { console.error("An error occurred:", error); } })(); ``` ``` from playwright.sync_api import sync_playwright def main(): try: with sync_playwright() as p: browser = p.chromium.launch( proxy={ "server": "socks5://proxy-saver.scrapfly.io:3333", } ) context = browser.new_context(ignore_https_errors=True) context.set_extra_http_headers({ "X-SAVER-AUTH": "proxyId-XXX:scp-live-XXX" }) page = context.new_page() page.goto("https://httpbin.dev/anything") html = page.content() print(html) except Exception as e: print("An error occurred:", e) finally: browser.close() if __name__ == "__main__": main() ``` ## UDP, SOCKS5-UDP and HTTP/3 (QUIC) Proxy Saver tunnels UDP, the transport behind HTTP/3 (QUIC), in addition to TCP. UDP datagrams are relayed through either of two inbound mechanisms: - **SOCKS5 UDP ASSOCIATE** (RFC 1928, section 7), for clients that speak SOCKS5 UDP. Chrome and Chromium do not support SOCKS5 UDP upstream, so use a Chromium build with SOCKS5-UDP enabled, or a SOCKS5-UDP-capable client. - **MASQUE CONNECT-UDP** (RFC 9298), over HTTP/3 on the instance's port (UDP), or over HTTP/2 and HTTP/1.1 with RFC 9297 capsules, for clients that tunnel UDP over HTTP. On the egress side, Proxy Saver reproduces the browser's **QUIC / HTTP/3 fingerprint** when replaying a profile: the QUIC ClientHello (JA4-QUIC, transport parameters and their wire order, GREASE), the HTTP/3 control-stream SETTINGS, GREASE frame and PRIORITY\_UPDATE, and the request pseudo-header order, so the upstream sees a byte-faithful browser rather than a generic QUIC client. Profiles carry this in their `Quic` and `Http3` sections alongside `Tls` and `Http2`. ### How your upstream proxy carries UDP HTTP/3 reaches the target only when the upstream proxy configured on the Proxy Saver instance carries UDP. The instance form offers three ways, depending on what your provider supports: | Upstream | TCP | UDP | |---|---|---| | Proxy, UDP **None** | the proxy | none: clients fall back to HTTP/2 | | SOCKS5 or SOCKS5H proxy, UDP **SOCKS5 UDP ASSOCIATE** | the proxy | UDP ASSOCIATE on the same proxy | | Proxy, UDP **MASQUE CONNECT-UDP** | the proxy | MASQUE CONNECT-UDP (RFC 9298, HTTP/3) to a separate endpoint, with the proxy's credentials | | **MASQUE-IP endpoint** | HTTP CONNECT over HTTP/3 | MASQUE CONNECT-UDP, same endpoint | The MASQUE endpoint is optional. Proxy Saver checks the upstream when you save the instance: first the UDP leg, with a full QUIC handshake through it, then the TCP leg, with an HTTPS request through the proxy. A TCP leg that fails blocks the save; a UDP leg that fails is saved with a warning (HTTP/3 then falls back to HTTP/2). A passed UDP check shows that the exit relayed a QUIC handshake on port 443; it does not test your targets. Credentials encrypted with your own key are not checked. Use **Check connectivity** on the instance page to test both legs again. ### Sessions, targeting and HTTP/3 Both legs use the instance's credentials plus the parameters you forward after `|`. Whether HTTP/3 works then depends on how your provider handles sessions: - A provider keys a sticky session on its session id and on every targeting parameter (country, city, OS and so on). Send exactly the same parameters on every TCP and UDP connection of a session, or they leave from different exits. Without a session parameter, TCP and UDP connections leave from different IPs. - Some providers bind a session to an exit at its first connection, and an exit bound by a TCP connection may relay no UDP. Open the session with its UDP (HTTP/3) connection first. If HTTP/3 keeps falling back to HTTP/2 on a session, start a new session id. - Some providers refuse UDP to some ports on the MASQUE leg, for example 53 (DNS) and 3478 (STUN). Resolve names remotely (SOCKS5H or DNS over HTTPS), and expect WebRTC STUN to fail. Bandwidth is metered and billed at the standard Proxy Saver rate regardless of transport.