# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification](https://scrapfly.io/docs/scrape-api/specification) - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification](https://scrapfly.io/docs/crawler-api/specification) - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification](https://scrapfly.io/docs/screenshot-api/specification) - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification](https://scrapfly.io/docs/extraction-api/specification) - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Bring Your Own Proxy](https://scrapfly.io/docs/cloud-browser-api/bring-your-own-proxy) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [Vibium](https://scrapfly.io/docs/cloud-browser-api/vibium) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Unblock API [ View as markdown ](https://scrapfly.io/?view=markdown) Copy for LLM Copy for LLM [ Open in ChatGPT ](https://chatgpt.com/?hints=search&prompt=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fcloud-browser-api%2Funblock%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Claude ](https://claude.ai/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fcloud-browser-api%2Funblock%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Perplexity ](https://www.perplexity.ai/search/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fcloud-browser-api%2Funblock%20so%20I%20can%20ask%20questions%20about%20it.) 1. [Cloud Browser](https://scrapfly.io/docs/cloud-browser-api/getting-started) 2. Unblock API The **Unblock API** automatically bypasses anti-bot protection for a target URL, then returns a WebSocket URL so you can connect with Playwright or Puppeteer to a browser session that's already past the protection. **Beta Feature:** Cloud Browser is currently in beta. ## How It Works The Unblock API combines Scrapfly's advanced anti-bot bypass technology with Cloud Browser's remote browser access: 1. **Request:** Send a POST request with your target URL and configuration 2. **Bypass:** Scrapfly allocates a browser, navigates to the URL with Anti Scraping Protection (ASP) enabled 3. **Handoff:** Once the page loads successfully, you receive a WebSocket URL 4. **Connect:** Use Playwright or Puppeteer to connect and interact with the unblocked page **Use Case:** Perfect for scraping sites with strong anti-bot protection where you need to interact with the page after bypassing the protection (clicking buttons, filling forms, scrolling, etc.). ## API Endpoint | **Method** | `POST` | |---|---| | **URL** | `https://browser.scrapfly.io/unblock` | | **Authentication** | Query parameter: `?key=YOUR_API_KEY` | | **Content-Type** | `application/json` | ### Request Parameters | Parameter | Type | Required | Default | Description | |---|---|---|---|---| | `url` | string | Yes | - | Target URL to navigate to and bypass protection | | `method` | string | No | `GET` | HTTP method: GET, POST, PUT, PATCH, DELETE | | `headers` | object | No | `{}` | Custom request headers as key-value pairs | | `body` | string | No | `""` | Request body for POST/PUT/PATCH requests | | `proxy_pool` | string | No | `public_residential_pool` | Proxy pool: `public_datacenter_pool` or `public_residential_pool` | | `country` | string | No | `""` (auto) | ISO country code for proxy geolocation (e.g., `us`, `gb`, `de`) | | `os` | string | No | `""` (random) | Operating system fingerprint: `linux`, `windows`, `macos` | | `timeout` | integer | No | `60` | Navigation timeout in seconds (max 300) | | `browser_timeout` | integer | No | `900` | Browser session timeout in seconds (max 1800) | ### Response Format On success, the API returns a JSON response with: ``` { "ws_url": "wss://browser.scrapfly.io/?key=YOUR_API_KEY&session=unblock-1234567890", "session_id": "unblock-1234567890", "run_id": "01HWXYZ..." } ``` | Field | Description | |---|---| | `ws_url` | WebSocket URL to connect with Playwright/Puppeteer | | `session_id` | Session identifier for reconnection and monitoring | | `run_id` | Unique run identifier for debugging and support | ## Basic Usage Select your preferred technology to see a complete example: Python Puppeteer Playwright JS cURL ``` import requests from playwright.sync_api import sync_playwright API_KEY = '' def unblock_and_scrape(): # Step 1: Call the Unblock API response = requests.post( f'https://browser.scrapfly.io/unblock?key={API_KEY}', json={ 'url': 'https://example.com/protected-page', 'proxy_pool': 'public_residential_pool', 'country': 'us' } ) data = response.json() ws_url = data['ws_url'] session_id = data['session_id'] print(f'Unblock successful! Session: {session_id}') # Step 2: Connect with Playwright with sync_playwright() as p: browser = p.chromium.connect_over_cdp(ws_url) context = browser.contexts[0] page = context.pages[0] if context.pages else context.new_page() # Step 3: Page is already loaded! print(f'Page title: {page.title()}') print(f'Page URL: {page.url}') # Scrape data, click buttons, fill forms... content = page.content() print(f'HTML length: {len(content)}') browser.close() unblock_and_scrape() ``` ``` const puppeteer = require('puppeteer-core'); const API_KEY = ''; async function unblockAndScrape() { // Step 1: Call the Unblock API const response = await fetch(`https://browser.scrapfly.io/unblock?key=${API_KEY}`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ url: 'https://example.com/protected-page', proxy_pool: 'public_residential_pool', country: 'us' }) }); const { ws_url, session_id } = await response.json(); console.log('Unblock successful! Session:', session_id); // Step 2: Connect to the unblocked browser const browser = await puppeteer.connect({ browserWSEndpoint: ws_url }); // Step 3: Get the page and interact with it const pages = await browser.pages(); const page = pages[0] || await browser.newPage(); // The page is already loaded and past anti-bot protection! const title = await page.title(); console.log('Page title:', title); // Scrape data, click buttons, fill forms... const content = await page.content(); console.log('Page HTML length:', content.length); await browser.close(); } unblockAndScrape(); ``` ``` const { chromium } = require('playwright'); const API_KEY = ''; async function unblockAndScrape() { // Step 1: Call the Unblock API const response = await fetch(`https://browser.scrapfly.io/unblock?key=${API_KEY}`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ url: 'https://example.com/protected-page', proxy_pool: 'public_residential_pool' }) }); const { ws_url, session_id } = await response.json(); console.log('Unblock successful! Session:', session_id); // Step 2: Connect via CDP const browser = await chromium.connectOverCDP(ws_url); const context = browser.contexts()[0]; const page = context.pages()[0] || await context.newPage(); // Step 3: Page is already loaded - interact with it! const title = await page.title(); console.log('Page title:', title); // Extract data const data = await page.evaluate(() => { return document.body.innerText; }); console.log('Page text:', data.substring(0, 200)); await browser.close(); } unblockAndScrape(); ``` First, call the Unblock API to get a WebSocket URL: ``` curl -X POST "https://browser.scrapfly.io/unblock?key=" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com/protected-page", "proxy_pool": "public_residential_pool", "country": "us" }' ``` Then connect to the returned `ws_url` using Playwright or Puppeteer. ## Advanced Usage ### Custom Headers Pass custom request headers for the initial navigation: ``` { "url": "https://example.com/api-page", "headers": { "Accept-Language": "en-US,en;q=0.9", "Accept": "text/html,application/xhtml+xml", "X-Custom-Header": "custom-value" } } ``` ### POST Requests Navigate to pages that require POST requests: ``` { "url": "https://example.com/login", "method": "POST", "headers": { "Content-Type": "application/x-www-form-urlencoded" }, "body": "username=user&password=pass" } ``` ### Geo-Targeting Use specific country proxies for geo-restricted content: ``` { "url": "https://example.com/regional-content", "proxy_pool": "public_residential_pool", "country": "gb" } ``` ### OS Fingerprint Specify the operating system for browser fingerprinting: ``` { "url": "https://example.com/page", "os": "windows" } ``` Available OS options: - `windows` - Windows fingerprint - `macos` - macOS fingerprint - `linux` - Linux fingerprint - `""` (empty) - Random selection (default) ## Session Management The Unblock API creates a browser session with `auto_close=false`, meaning the browser stays alive after the initial navigation completes. You can: - **Reconnect:** Use the same `session_id` to reconnect to the browser - **Monitor:** View the session in the Cloud Browser dashboard - **Debug:** Use [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) for visual debugging **Important:** Always call `browser.close()` when finished to terminate the session and stop billing. Sessions are billed until explicitly closed or they reach the `browser_timeout` limit. ### Reconnecting to Sessions ``` // Reconnect using the ws_url from the unblock response const browser = await puppeteer.connect({ browserWSEndpoint: ws_url // Same URL from unblock response }); // Or construct manually with session_id const reconnectUrl = `wss://browser.scrapfly.io/?key=${API_KEY}&session=${session_id}`; const browser2 = await puppeteer.connect({ browserWSEndpoint: reconnectUrl }); ``` ## Error Handling The Unblock API may return these error codes: | Error Code | HTTP Status | Description | |---|---|---| | `ERR::BROWSER::CONFIG_ERROR` | 400 | Invalid configuration (missing URL, invalid parameters) | | `ERR::BROWSER::ALLOCATION_FAILED` | 503 | Failed to allocate browser (capacity issue) | | `ERR::BROWSER::NAVIGATION_TIMEOUT` | 504 | Page navigation timed out | | `ERR::BROWSER::UNBLOCK_FAILED` | 503 | Failed to bypass anti-bot protection | | `ERR::BROWSER::TOO_MANY_CONCURRENT_REQUEST` | 429 | Concurrency limit exceeded | ### Error Response Format ``` { "error_id": "ERR::BROWSER::NAVIGATION_TIMEOUT", "message": "Page load timed out after 60 seconds", "http_code": 504, "is_retryable": true } ``` ### Retry Strategy ``` async function unblockWithRetry(url, maxRetries = 3) { for (let attempt = 1; attempt <= maxRetries; attempt++) { try { const response = await fetch(`https://browser.scrapfly.io/unblock?key=${API_KEY}`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ url }) }); if (!response.ok) { const error = await response.json(); if (error.is_retryable && attempt < maxRetries) { console.log(`Attempt ${attempt} failed, retrying...`); await new Promise(r => setTimeout(r, 2000 * attempt)); // Exponential backoff continue; } throw new Error(error.message); } return await response.json(); } catch (error) { if (attempt === maxRetries) throw error; } } } // Usage const result = await unblockWithRetry('https://example.com/protected'); console.log('Success:', result.ws_url); ``` ## Billing The Unblock API is billed the same as regular Cloud Browser sessions: - Session time is billed per 30 seconds (rounded up) - Bandwidth is billed separately - Residential proxy pool has higher cost than datacenter See [Cloud Browser Billing](https://scrapfly.io/docs/cloud-browser-api/billing) for detailed pricing. ## Unblock API vs Regular Connection | Feature | Unblock API | Regular WebSocket Connection | |---|---|---| | Anti-bot bypass | Automatic | Manual (via ASP parameter) | | Initial navigation | Handled by API | You handle it | | Page state on connect | Page already loaded | Empty browser | | Use case | Post-bypass interaction | Full browser control | ## Related Documentation - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - Introduction to Cloud Browser API - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - Reconnect to browser sessions - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - Visual debugging for browser sessions - [Puppeteer Integration](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - Puppeteer-specific documentation - [Playwright Integration](https://scrapfly.io/docs/cloud-browser-api/playwright) - Playwright-specific documentation - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - Understand session costs - [Error Reference](https://scrapfly.io/docs/cloud-browser-api/errors) - Full error code reference