# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Batch (Multi-URL Scraping)](https://scrapfly.io/docs/scrape-api/batch) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Schedule](https://scrapfly.io/docs/scrape-api/schedule) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Schedule](https://scrapfly.io/docs/crawler-api/schedule) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Schedule](https://scrapfly.io/docs/screenshot-api/schedule) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Data API #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [Captcha Solver](https://scrapfly.io/docs/cloud-browser-api/captcha-solver) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) - [Native Browser MCP](https://scrapfly.io/docs/cloud-browser-api/mcp) - [DevTools Protocol](https://scrapfly.io/docs/cloud-browser-api/cdp-reference) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Proxy Saver Protocols The Proxy Saver supports all standard and modern proxy protocols, ensuring compatibility with any browser or web scraping tool. Below are the details and specificities of each protocol's usage. > While all the provided examples account for the possibility that the Proxy Saver certificates may not be properly installed, the recommended approach is to [configure them correctly](https://scrapfly.io/docs/proxy-saver/certificates) rather than bypassing SSL errors. ## HTTP HTTP is the simplest proxy protocol and is universally compatible with all browsers and libraries. Below are some examples: - [Curl](#basic-use-curl-http) - [Playwright (js)](#scrape-http-proxy-playwright-js) - [Playwright (python)](#scrape-http-proxy-playwright-python) ``` curl \ --insecure \ --proxy http://proxyId-XXX:scp-live-XXX@proxy-saver.scrapfly.io:3333 \ https://httpbin.dev/anything ``` ``` const {chromium} = require("playwright"); (async () => { try { const browser = await chromium.launch({ proxy: { server: 'http://proxy-saver.scrapfly.io:3333', username: 'proxyId-XXX', password: 'scp-live-XXX' } }); const context = await browser.newContext({ignoreHTTPSErrors: true}); const page = await context.newPage(); await page.goto("https://httpbin.dev/anything"); const html = await page.content(); console.log(html); await browser.close(); } catch (error) { console.error("An error occurred:", error); } })(); ``` ``` from playwright.sync_api import sync_playwright def main(): try: with sync_playwright() as p: browser = p.chromium.launch( proxy={ "server": "http://proxy-saver.scrapfly.io:3333", "username": "proxyId-XXX", "password": "scp-live-XXX", } ) context = browser.new_context(ignore_https_errors=True) page = context.new_page() page.goto("https://httpbin.dev/anything") html = page.content() print(html) except Exception as e: print("An error occurred:", e) finally: browser.close() if __name__ == "__main__": main() ``` ## HTTPS / HTTP2 Using HTTP as a proxy protocol is straightforward, but opting for HTTPS ensures that your credentials are securely encrypted from the very start of the CONNECT phase. By default, curl will use HTTP/1.1 over TLS unless you have a recent version that supports HTTP/2 and explicitly specify the necessary flags. Chrome, on the other hand, defaults to HTTP/2 but will gracefully downgrade to HTTP/1.1 over TLS if needed. The Proxy Saver natively supports HTTP/2 while also maintaining compatibility with HTTP/1.1 over TLS to accommodate the broadest range of browser implementations. - [Curl (HTTPS)](#basic-use-curl-https) - [Curl (HTTP2)](#basic-use-curl-http2) - [Playwright (js)](#scrape-https-proxy-playwright-js) - [Playwright (python)](#scrape-https-proxy-playwright-python) ``` curl \ --insecure --proxy-insecure \ --proxy https://proxyId-XXX:scp-live-XXX@proxy-saver.scrapfly.io:3333 \ https://httpbin.dev/anything ``` ``` curl \ --insecure --proxy-insecure \ --proxy-http2 \ --proxy https://proxyId-XXX:scp-live-XXX@proxy-saver.scrapfly.io:3333 \ https://httpbin.dev/anything ``` ``` const {chromium} = require("playwright"); (async () => { try { const browser = await chromium.launch({ proxy: { server: 'https://proxy-saver.scrapfly.io:3333', username: 'proxyId-XXX', password: 'scp-live-XXX' }, args: ['--ignore-certificate-errors'], }); const context = await browser.newContext({ignoreHTTPSErrors: true}); const page = await context.newPage(); await page.goto("https://httpbin.dev/anything"); const html = await page.content(); console.log(html); await browser.close(); } catch (error) { console.error("An error occurred:", error); } })(); ``` ``` from playwright.sync_api import sync_playwright def main(): try: with sync_playwright() as p: browser = p.chromium.launch( proxy={ "server": "https://proxy-saver.scrapfly.io:3333", "username": "proxyId-XXX", "password": "scp-live-XXX", }, args=["--ignore-certificate-errors"] ) context = browser.new_context(ignore_https_errors=True) page = context.new_page() page.goto("https://httpbin.dev/anything") html = page.content() print(html) except Exception as e: print("An error occurred:", e) finally: browser.close() if __name__ == "__main__": main() ``` ## SOCKS5 Socks5/h is another popular protocol used for HTTP proxying ### Socks5 auth mechanism For browsers and libraries that support native SOCKS5 authentication, the Proxy Saver does as well, ensuring seamless integration. - [Curl (SOCKS5 Auth)](#basic-use-curl-https) ``` curl \ --insecure \ --proxy socks5h://proxyId-XXX:scp-live-XXX@proxy-saver.scrapfly.io:3333 \ https://httpbin.dev/anything ``` However, not all browsers support this mechanism. > While Chrome and Chromium do not support the SOCKS5 authentication mechanism (and have no plans to), Scrapfly offers a reliable workaround. For browsers and libraries that do not support SOCKS5 authentication natively, Scrapfly offers a simple workaround using a custom header. Simply include your authentication string in the `X-SAVER-AUTH` header. This custom authentication mechanism enables the use of the SOCKS5 protocol with any library that supports it. Below are some usage examples: - [Curl (X-SAVER-AUTH)](#basic-use-curl-socks5h-x-auth) - [Playwright (js)](#scrape-socks5-proxy-playwright-js) - [Playwright (python)](#scrape-socks5-proxy-playwright-python) ``` curl \ --insecure \ --proxy socks5h://proxy-saver.scrapfly.io:3333 \ --header "X-SAVER-AUTH: proxyId-XXX:scp-live-XXX" \ https://httpbin.dev/anything ``` ``` const { chromium } = require("playwright"); const xSaverAuth = { 'X-SAVER-AUTH': 'proxyId-XXX:scp-live-XXX'}; (async () => { try { const browser = await chromium.launch({ proxy: { server: 'socks5://proxy-saver.scrapfly.io:3333', } }); const context = await browser.newContext({ ignoreHTTPSErrors: true }); await context.setExtraHTTPHeaders(xSaverAuth); const page = await context.newPage(); await page.goto("https://httpbin.dev/anything"); const html = await page.content(); console.log(html); await browser.close(); } catch (error) { console.error("An error occurred:", error); } })(); ``` ``` from playwright.sync_api import sync_playwright def main(): try: with sync_playwright() as p: browser = p.chromium.launch( proxy={ "server": "socks5://proxy-saver.scrapfly.io:3333", } ) context = browser.new_context(ignore_https_errors=True) context.set_extra_http_headers({ "X-SAVER-AUTH": "proxyId-XXX:scp-live-XXX" }) page = context.new_page() page.goto("https://httpbin.dev/anything") html = page.content() print(html) except Exception as e: print("An error occurred:", e) finally: browser.close() if __name__ == "__main__": main() ``` ## UDP, SOCKS5-UDP and HTTP/3 (QUIC) Proxy Saver tunnels UDP, the transport behind HTTP/3 (QUIC), in addition to TCP. UDP datagrams are relayed through either of two inbound mechanisms: - **SOCKS5 UDP ASSOCIATE** (RFC 1928, section 7), for clients that speak SOCKS5 UDP. Chrome and Chromium do not support SOCKS5 UDP upstream, so use a Chromium build with SOCKS5-UDP enabled, or a SOCKS5-UDP-capable client. - **MASQUE CONNECT-UDP** (RFC 9298, with RFC 9297 capsules over HTTP/2), for clients that tunnel UDP over HTTP. On the egress side, Proxy Saver reproduces the browser's **QUIC / HTTP/3 fingerprint** when replaying a profile: the QUIC ClientHello (JA4-QUIC, transport parameters and their wire order, GREASE), the HTTP/3 control-stream SETTINGS, GREASE frame and PRIORITY\_UPDATE, and the request pseudo-header order, so the upstream sees a byte-faithful browser rather than a generic QUIC client. Profiles carry this in their `Quic` and `Http3` sections alongside `Tls` and `Http2`. Bandwidth is metered and billed at the standard Proxy Saver rate regardless of transport.