# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Bring Your Own Proxy](https://scrapfly.io/docs/cloud-browser-api/bring-your-own-proxy) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Stagehand Integration [ View as markdown ](https://scrapfly.io/?view=markdown) Copy for LLM Copy for LLM [ Open in ChatGPT ](https://chatgpt.com/?hints=search&prompt=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fcloud-browser-api%2Fstagehand%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Claude ](https://claude.ai/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fcloud-browser-api%2Fstagehand%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Perplexity ](https://www.perplexity.ai/search/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fcloud-browser-api%2Fstagehand%20so%20I%20can%20ask%20questions%20about%20it.) 1. [Cloud Browser](https://scrapfly.io/docs/cloud-browser-api/getting-started) 2. Stagehand [Stagehand](https://github.com/browserbase/stagehand) is an AI-powered browser automation framework that combines natural language instructions with code-based precision. Connect it to Scrapfly Cloud Browser for intelligent, self-healing automation with built-in proxies and fingerprinting. **Beta Feature:** Cloud Browser is currently in beta. ## What is Stagehand? Stagehand is a browser automation framework that uses AI to understand and interact with web pages using natural language. Built on Playwright, it offers: - **Natural Language Control** - Describe actions in plain English instead of writing complex selectors - **Self-Healing Automation** - AI adapts when website structures change - **Smart Caching** - Reduces AI costs by caching repeated actions - **Hybrid Approach** - Mix AI actions with traditional code for reliability ## Installation Install Stagehand via npm: ``` npm install @browserbasehq/stagehand ``` > Stagehand requires an LLM API key (OpenAI, Anthropic, etc.) for AI-powered actions. Set `OPENAI_API_KEY` or configure your preferred provider. ## Quick Start with AI Actions Connect Stagehand to Cloud Browser and use AI to interact with pages: ``` import { Stagehand } from "@browserbasehq/stagehand"; const API_KEY = ''; const BROWSER_WS = `wss://' ~ public_cloud_browser_endpoint ~ '?api_key=${API_KEY}&proxy_pool=public_datacenter_pool&os=linux`; async function run() { // Initialize Stagehand with Cloud Browser via CDP const stagehand = new Stagehand({ env: "LOCAL", localBrowserLaunchOptions: { cdpUrl: BROWSER_WS, }, }); await stagehand.init(); // Get a page and navigate const page = await stagehand.ctx.newPage(); await page.goto('https://web-scraping.dev'); // Use AI to perform actions with natural language await stagehand.act("click on the products link"); // Extract data using AI const products = await stagehand.extract({ instruction: "extract all product names and prices", schema: { products: [{ name: "string", price: "string" }] } }); console.log('Products:', products); await stagehand.close(); } run(); ``` ## AI-Powered Scraping Use Stagehand's AI capabilities to scrape complex, dynamic websites without writing selectors: ``` import { Stagehand } from "@browserbasehq/stagehand"; import { z } from "zod"; const API_KEY = ''; const BROWSER_WS = `wss://' ~ public_cloud_browser_endpoint ~ '?api_key=${API_KEY}&proxy_pool=public_datacenter_pool`; async function scrapeWithAI() { const stagehand = new Stagehand({ env: "LOCAL", localBrowserLaunchOptions: { cdpUrl: BROWSER_WS }, }); await stagehand.init(); const page = await stagehand.ctx.newPage(); await page.goto('https://web-scraping.dev/products'); // Define extraction schema with Zod const ProductSchema = z.object({ products: z.array(z.object({ title: z.string(), price: z.string(), rating: z.number().optional(), })) }); // AI extracts data matching the schema const result = await stagehand.extract({ instruction: "extract all products with their title, price, and rating", schema: ProductSchema, }); console.log('Extracted products:', result.products); await stagehand.close(); return result; } scrapeWithAI(); ``` ## Natural Language Interactions Handle complex workflows like login flows using natural language instead of brittle selectors: ``` import { Stagehand } from "@browserbasehq/stagehand"; import { z } from "zod"; const API_KEY = ''; const BROWSER_WS = `wss://' ~ public_cloud_browser_endpoint ~ '?api_key=${API_KEY}&proxy_pool=public_datacenter_pool`; async function login() { const stagehand = new Stagehand({ env: "LOCAL", localBrowserLaunchOptions: { cdpUrl: BROWSER_WS }, }); await stagehand.init(); const page = await stagehand.ctx.newPage(); await page.goto('https://web-scraping.dev/login'); // Use natural language to interact with the login form await stagehand.act("fill in the email field with user@example.com"); await stagehand.act("fill in the password field with mypassword"); await stagehand.act("click the login button"); // Wait for login to complete await page.waitForURL('**/dashboard'); // Verify login success with AI const isLoggedIn = await stagehand.extract({ instruction: "check if the user is logged in", schema: z.object({ loggedIn: z.boolean(), username: z.string().optional() }) }); console.log('Login successful:', isLoggedIn); await stagehand.close(); } login(); ``` ## Multi-Step AI Agent Use Stagehand's autonomous agent mode to complete complex, multi-step tasks: ``` import { Stagehand } from "@browserbasehq/stagehand"; const API_KEY = ''; const BROWSER_WS = `wss://' ~ public_cloud_browser_endpoint ~ '?api_key=${API_KEY}&proxy_pool=public_residential_pool`; async function autonomousAgent() { const stagehand = new Stagehand({ env: "LOCAL", localBrowserLaunchOptions: { cdpUrl: BROWSER_WS }, }); await stagehand.init(); const page = await stagehand.ctx.newPage(); await page.goto('https://web-scraping.dev'); // Agent autonomously completes the task const result = await stagehand.agent({ instruction: ` 1. Find all product categories 2. Click on the first category 3. Extract the top 5 products with prices 4. Return the results ` }); console.log('Agent result:', result); await stagehand.close(); } autonomousAgent(); ``` ## Hybrid: AI + Traditional Code Combine AI actions with traditional Playwright code for the best of both worlds: ``` import { Stagehand } from "@browserbasehq/stagehand"; const API_KEY = ''; const BROWSER_WS = `wss://' ~ public_cloud_browser_endpoint ~ '?api_key=${API_KEY}&proxy_pool=public_datacenter_pool`; async function hybridAutomation() { const stagehand = new Stagehand({ env: "LOCAL", localBrowserLaunchOptions: { cdpUrl: BROWSER_WS }, }); await stagehand.init(); // Traditional Playwright code for known, predictable steps await page.goto('https://web-scraping.dev/products'); await page.waitForSelector('.product'); // Switch to AI for complex or changing UI await stagehand.act("sort products by price from high to low"); await stagehand.act("click on the most expensive product"); // Back to traditional code for data extraction const title = await page.locator('h1').textContent(); const price = await page.locator('.price').textContent(); console.log(`Product: ${title}, Price: ${price}`); await stagehand.close(); } hybridAutomation(); ``` ## Session Persistence Maintain browser state across Stagehand connections using the `session` parameter: ``` import { Stagehand } from "@browserbasehq/stagehand"; const API_KEY = ''; const SESSION_ID = 'my-ai-session'; // First connection: Login with AI async function firstConnection() { const stagehand = new Stagehand({ env: "LOCAL", localBrowserLaunchOptions: { cdpUrl: `wss://' ~ public_cloud_browser_endpoint ~ '?api_key=${API_KEY}&session=${SESSION_ID}`, }, }); await stagehand.init(); const page = await stagehand.ctx.newPage(); await page.goto('https://web-scraping.dev/login'); await stagehand.act("log in with email user@example.com and password mypassword"); await stagehand.close(); // Session is preserved } // Second connection: Reuse the session async function secondConnection() { const stagehand = new Stagehand({ env: "LOCAL", localBrowserLaunchOptions: { cdpUrl: `wss://' ~ public_cloud_browser_endpoint ~ '?api_key=${API_KEY}&session=${SESSION_ID}`, }, }); await stagehand.init(); const page = await stagehand.ctx.newPage(); await page.goto('https://web-scraping.dev/dashboard'); // Already logged in from previous session! const data = await stagehand.extract({ instruction: "get user account information" }); console.log('User data:', data); await stagehand.close(); } ``` ## Proxy Options | Proxy Pool | Use Case | Cost | |---|---|---| | `datacenter` | General scraping, high speed, lower cost | 1 credits/30s + 7 credits/MB | | `residential` | Protected sites, geo-targeting, AI-powered bypass | 1 credits/30s + 52 credits/MB | ## Best Practices - **Set LLM API key** - Configure `OPENAI_API_KEY` or your preferred AI provider - **Use caching** - Stagehand automatically caches AI actions to reduce costs - **Mix AI and code** - Use AI for complex/changing UI, code for predictable steps - **Close browsers** - Always call `stagehand.close()` to stop billing - **Preview actions** - Use Stagehand's preview mode in development to verify AI actions - **Handle errors** - Wrap AI actions in try/catch for robust automation ## Cost Optimization Stagehand automation combines Cloud Browser costs with LLM API costs: - **Cloud Browser** - Billed per time + bandwidth (see table above) - **LLM API** - Billed by your AI provider (OpenAI, Anthropic, etc.) - **Caching** - Stagehand caches actions automatically to minimize LLM calls - **Hybrid approach** - Use traditional code where possible to reduce AI costs **Tip:** Stagehand's smart caching can reduce LLM costs by up to 90% for repeated workflows. ## Related - [Cloud Browser Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Billing & Pricing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Playwright Integration](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Puppeteer Integration](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Stagehand GitHub](https://github.com/browserbase/stagehand) - [Stagehand Documentation](https://docs.stagehand.dev)