# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Unblocker (formerly ASP)](https://scrapfly.io/docs/scrape-api/unblocker) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Batch (Multi-URL Scraping)](https://scrapfly.io/docs/scrape-api/batch) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Schedule](https://scrapfly.io/docs/scrape-api/schedule) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Search](https://scrapfly.io/docs/crawler-api/search) - [Prompt & Extract](https://scrapfly.io/docs/crawler-api/prompt) - [Auto Refresh](https://scrapfly.io/docs/crawler-api/refresh) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Schedule](https://scrapfly.io/docs/crawler-api/schedule) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Schedule](https://scrapfly.io/docs/screenshot-api/schedule) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Data API #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [Captcha Solver](https://scrapfly.io/docs/cloud-browser-api/captcha-solver) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) - [Credential Vault](https://scrapfly.io/docs/cloud-browser-api/credential-vault) - [Native Browser MCP](https://scrapfly.io/docs/cloud-browser-api/mcp) - [DevTools Protocol](https://scrapfly.io/docs/cloud-browser-api/cdp-reference) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [1Password](https://scrapfly.io/docs/cloud-browser-api/1password) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Projects and Environments Scrapfly [Projects](https://scrapfly.io/dashboard/project) feature is an easy way to organize Scrapfly resources. This feature is ideal for managing multiple web scraping projects with its own budgets, quotas and configurations. Each project gives you two dedicated environments: **LIVE** and **TEST**. Each environment has a dedicated API key. The goal of each environment is to separate the production and sandbox. So you can confidently work/test/try under **TEST** without worrying about production. Everyone start with a **"default"** project which cannot be deleted and more projects can be created on demand. All projects can be managed, created and modified using the [Projects Dashboard](https://scrapfly.io/dashboard/project) 👉 web interface for managing projects > **The Number of Available Projects Depends on Your Plan** > > | FREE | DISCOVERY | PRO | STARTUP | ENTERPRISE | > |---|---|---|---|---| > | 1 projects | 2 projects | 5 projects | 15 projects | 50 projects | [See in Your dashboard](https://scrapfly.io/dashboard) ## API Keys and Key Reset Each project has two dedicated API keys for **LIVE** and **TEST** environments and can be always found on the top part of your scrapfly dashboard. For invalidating your API keys visit the [Projects Dashboard](https://scrapfly.io/dashboard/project) and select `Actions->Regenerate API Key` option. Alternatively, the keys can be reset in each individual project's control panel which you can access by clicking on the project name. ## Scopes All the resources listed below are scoped per project and per environment: - Monitoring - Webhook - Schedule - Throttling - Integration ## Authentication & Usage #### Web Platform The project selector is available in most pages of the web interface. It allows you to select the project and environment you want to work on. The [projects page](https://scrapfly.io/dashboard/project) allows to create, manage and delete projects. > Deleted projects are not recoverable - all of the data will be lost! #### API Each project and environment has a dedicated API key so to switch between projects and environments just change the used key. ## Quota Each project can have can be configured with a quota. Setting quota is a way to limit the project usage for aspects like: - API credit limit - Budget limit - Concurrency limit - Whether extra API credit use is allowed ## Security A project can accept requests only from the IP addresses and CIDR ranges you list. ### Authorized IP address or CIDR Open [Projects](https://scrapfly.io/dashboard/project), click **Edit Settings** on your project, tab **Security & Access**, field **Authorized IP address or CIDR**. It accepts IPv4, IPv6 and CIDR, separated by `,`. While it is empty every address is accepted. A change takes up to a minute to apply, up to three minutes for Proxy Saver. A request from any other address is refused, even with a valid API key. The API answers `401` `Network Unauthorized` and names the address it checked, Proxy Saver answers `403` `ERR_PROJECT_NETWORK_WHITELIST`, and the MCP server answers `401` with the address it saw and a link to the project settings. ### Allow an AI provider or an automation platform The restriction applies to the address the request comes from. A hosted AI client or automation platform calls Scrapfly from its provider's servers, not from your machine, so allowing only your own IP blocks it. This includes Claude Desktop custom connectors: the app runs on your computer, the call comes from Anthropic. Add the ranges the provider publishes. Clearing the field also works, but every address is then accepted and the API key is the only protection. If a project does production scraping, prefer a separate project, or an API key limited to the products the client needs, over loosening that project's restriction. A client that runs on your machine (Cursor, VS Code, Cline, Windsurf, Zed, Claude Code with a server added by `claude mcp add`, your own scripts) calls from your own public IP. The cloud sessions of those tools (Cursor Cloud Agents, Claude Code on the web) and remote workspaces (VS Code Remote, Dev Containers, Codespaces) call from that environment's IP. **Scrapfly MCP server.** It checks the address of your AI client, then calls the Scrapfly API itself, and the API checks the MCP server's address. When a tool call fails with `Network Unauthorized`, the error names that address: add it too, next to the provider's ranges. | Provider | Product that calls Scrapfly | IP ranges | |---|---|---| | Anthropic | Claude custom connectors, Claude API MCP connector | [Claude IP addresses](https://platform.claude.com/docs/en/api/ip-addresses) (outbound) | | OpenAI | ChatGPT apps, connectors and GPT Actions (`chatgpt-connectors.json`); Codex cloud and ChatGPT agent use `chatgpt-agents.json` | [OpenAI IP egress ranges](https://developers.openai.com/api/docs/guides/ip-addresses) | | Microsoft | Copilot Studio and Power Automate connectors | [Managed connectors outbound IP addresses](https://learn.microsoft.com/en-us/connectors/common/outbound-ip-addresses) (Azure service tags `AzureConnectors.` and `PowerPlatformPlex`) | | n8n | n8n Cloud (self-hosted n8n calls from your own server) | [Find your IP addresses](https://docs.n8n.io/deploy/use-n8n-cloud/configure-cloud/find-your-ip-addresses) | | Make | Make scenarios | [Allow connections to and from Make IP addresses](https://help.make.com/allow-connections-to-and-from-make-ip-addresses) (listed per zone, use the one of your Make URL) | | Zapier | Zaps | [Zapier IP addresses and allowlisting](https://help.zapier.com/hc/en-us/articles/8496216020877-Zapier-IP-addresses-and-allowlisting) (no fixed range; [static IP](https://help.zapier.com/hc/en-us/articles/15406083674509-Use-a-static-IP-address-to-connect-to-Zapier) on Professional, Team and Enterprise plans, only for the apps Zapier lists) | | Vapi | Vapi tools and webhooks | [Static IP addresses](https://docs.vapi.ai/security-and-privacy/static-ip-addresses) (opt-in: `staticIpAddressesEnabled` on the server object) | > Each provider changes its ranges, so this page links to them instead of copying them. If calls that worked start failing with a `401`, check the provider's page again. Anyone with an account at a provider calls from the same ranges, so once you allow them the restriction no longer stops a leaked key from being used through that provider: keep the key private and rotate it if it leaks.