# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Batch (Multi-URL Scraping)](https://scrapfly.io/docs/scrape-api/batch) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Schedule](https://scrapfly.io/docs/scrape-api/schedule) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Schedule](https://scrapfly.io/docs/crawler-api/schedule) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Schedule](https://scrapfly.io/docs/screenshot-api/schedule) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Data API #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [Captcha Solver](https://scrapfly.io/docs/cloud-browser-api/captcha-solver) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) - [Native Browser MCP](https://scrapfly.io/docs/cloud-browser-api/mcp) - [DevTools Protocol](https://scrapfly.io/docs/cloud-browser-api/cdp-reference) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Scrapfly Agent Skills [Scrapfly Skills](https://github.com/scrapfly/skills) enhance the experience of using Scrapfly with AI agents. They act as documentation for your agent, providing the procedural knowledge it needs to effectively scrape web pages, extract data, capture screenshots, crawl websites, and automate browsers using the Scrapfly API. Install Scrapfly Skills with a single command to teach your agent how to use Scrapfly. Supports **40+ AI agents** including Claude Code, Cursor, Cline, Windsurf, GitHub Copilot, OpenAI Codex, Gemini, Roo, Kiro CLI, and many more. > Skills are like documentation for AI agents. Just as developers read docs to learn how to use an API, skills teach agents how to use tools effectively. Learn more at [skills.sh](https://skills.sh). ## Installation To get started, grab your API key from the [dashboard](https://scrapfly.io/dashboard). Then install all Scrapfly skills at once: ``` # Install every Scrapfly skill into every detected agent npx skills add scrapfly/skills ``` Or install individual skills depending on your needs (use `-s `, repeatable): ``` # Install individual skills (use -s , repeatable) npx skills add scrapfly/skills -s scrapfly-agent-rules npx skills add scrapfly/skills -s scrapfly-alerting npx skills add scrapfly/skills -s scrapfly-browser npx skills add scrapfly/skills -s scrapfly-cli npx skills add scrapfly/skills -s scrapfly-crawler npx skills add scrapfly/skills -s scrapfly-extraction npx skills add scrapfly/skills -s scrapfly-scraper npx skills add scrapfly/skills -s scrapfly-screenshot npx skills add scrapfly/skills -s scrapfly-sdk-go npx skills add scrapfly/skills -s scrapfly-sdk-rust npx skills add scrapfly/skills -s scrapfly-sdk-typescript # Target specific agents (claude-code, codex, cursor, gemini-cli, ...) npx skills add scrapfly/skills -a claude-code -a cursor # Install globally (~//skills/) instead of in the current project npx skills add scrapfly/skills --global ``` ## Setup After installing skills, set your Scrapfly API key as an environment variable: ``` # Set your Scrapfly API key export SCRAPFLY_API_KEY="your-api-key" ``` You can get your API key from the [Scrapfly dashboard](https://scrapfly.io/dashboard). Once configured, your AI agent will know how to use Scrapfly to scrape web pages, extract data, take screenshots, crawl websites, and automate browsers. ## Available Skills Scrapfly ships skills for every product: SDK skills for Python, TypeScript, Go, and Rust, a CLI skill for shell-tool agents, and a cross-tool rules skill for agents wired to the Scrapfly MCP server. | Skill | When to use | |---|---| | `scrapfly-agent-rules` | Cross-tool golden rules for an autonomous Scrapfly web agent connected to the Scrapfly MCP server (web\_scrape, screenshot, extract, classify\_block, the Cloud Browser lifecycle, snapshot/click/fill/type/press/scroll/select/drag, and WebMCP). Use when an LLM agent has access to the Scrapfly MCP toolset and needs to choose between stateless scraping and a stateful Cloud Browser session, decide when to call browser\_unblock, or coordinate multi-tool flows. | | `scrapfly-alerting` | Threshold-based alerting on Scrapfly account metrics. Wire up rules like "alert me when scrape success rate drops below 95% for 10 minutes" and receive notifications via email, webhook, or in-app. Use when the user asks to "set up an alert", "monitor my scrape success rate", "tell me when my scrapes start failing", "alert on ASP block rate", "watch a specific domain", or any task involving the Scrapfly alerting product. Covers the three call paths: REST API, MCP `alert\_\*` tools, and the `scrapfly alert` CLI subcommand. Enterprise plan feature; GATED behind the ALERTING feature flag on customer accounts. | | `scrapfly-browser` | Automate cloud browsers using the Scrapfly Cloud Browser API with Python Playwright | | `scrapfly-cli` | Use the Scrapfly CLI (`scrapfly`) to scrape web pages, capture screenshots, extract structured data with AI, crawl entire sites, and drive a cloud browser over CDP from a shell. JSON envelope on stdout for every command. Use when the user asks to "scrape a URL", "screenshot a page", "extract data from HTML", "crawl a site", "drive a browser with Playwright/CDP", "bypass anti-bot", or any task involving the `scrapfly` command. | | `scrapfly-crawler` | Crawl entire websites using the Scrapfly Crawler API with the Python SDK | | `scrapfly-extraction` | Extract structured data from web content using the Scrapfly Extraction API with the Python SDK | | `scrapfly-scraper` | Web scraping using the Scrapfly Scraper API with the Python SDK | | `scrapfly-screenshot` | Capture web page screenshots using the Scrapfly Screenshot API with the Python SDK | | `scrapfly-sdk-go` | Web scraping from Go with the Scrapfly SDK (`github.com/scrapfly/go-scrapfly`). Use when the user asks to scrape a URL, bypass anti-bot protection, render JavaScript, capture screenshots, or extract structured data from a Go program. | | `scrapfly-sdk-rust` | Web scraping from Rust with the Scrapfly SDK (`scrapfly-sdk` crate). Use when the user asks to scrape a URL, bypass anti-bot protection, render JavaScript, capture screenshots, or extract structured data from an async Rust program. | | `scrapfly-sdk-typescript` | Web scraping from TypeScript or JavaScript with the Scrapfly SDK (`scrapfly-sdk`). Use when the user asks to scrape a URL, bypass anti-bot protection, render JavaScript, capture screenshots, or extract structured data from Node.js, Deno, or Bun. | ## Supported Agents Scrapfly Skills are built on the open [Skills](https://skills.sh) ecosystem and support **40+ AI agents**, including: - Claude Code - Cursor - Cline - Windsurf - GitHub Copilot - VS Code - OpenAI Codex - Gemini - Roo - Kiro CLI - Goose - Trae - AMP - Antigravity - Droid - Kilo - OpenCode - And many more... For the full list of supported agents, see the [Skills directory](https://skills.sh). ## Related - [Building with AI](https://scrapfly.io/docs/integration/building-with-ai): agent skills, MCP, SDKs, and llms.txt in one place - [Scrapfly MCP Server](https://scrapfly.io/docs/mcp/getting-started): use Scrapfly through the Model Context Protocol - [Scrapfly SDK](https://scrapfly.io/docs/sdk): use Scrapfly directly from Python, TypeScript, Go, or Rust - [GitHub Repository](https://github.com/scrapfly/skills): view source code and contribute ## Pricing No additional costs. Scrapfly Skills use the same API credits as direct API usage.