# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Bring Your Own Proxy](https://scrapfly.io/docs/cloud-browser-api/bring-your-own-proxy) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) - [Native Browser MCP](https://scrapfly.io/docs/cloud-browser-api/mcp) - [DevTools Protocol](https://scrapfly.io/docs/cloud-browser-api/cdp-reference) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # MCP Authentication [ View as markdown ](https://scrapfly.io/?view=markdown) Copy for LLM Copy for LLM [ Open in ChatGPT ](https://chatgpt.com/?hints=search&prompt=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fauthentication%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Claude ](https://claude.ai/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fauthentication%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Perplexity ](https://www.perplexity.ai/search/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fauthentication%20so%20I%20can%20ask%20questions%20about%20it.) The Scrapfly MCP Server supports three authentication methods, each optimized for different security requirements and deployment scenarios. **New to MCP?** Start with our [Getting Started Guide](https://scrapfly.io/docs/mcp/getting-started) to understand the basics before diving into authentication methods. ## Quick Comparison | Feature | Query Parameter | Authorization Header | OAuth2 | |---|---|---|---| | Setup Complexity | Easiest | Moderate | Simple | | Security Level | Basic | Good | Best | | Key in Config Files | Yes | Via Env Vars | No | | Version Control Safe | No | With Env Vars | Yes | | Automatic Rotation | No | No | Yes | | Instant Revocation | Key Change Required | Key Change Required | Yes | | Multi-User Support | Shared Key | Shared Key | Individual Auth | | Audit Trail | Limited | Limited | Full | ### When to Use Each Method | Scenario | Recommended Method | |---|---| | Personal laptop, local development | Query Parameter | | Custom Python/Node.js MCP client | Authorization Header | | Docker container, cloud deployment | Authorization Header | | CI/CD pipeline, automated workflows | Authorization Header | | Remote MCP server (accessible by others) | OAuth2 | | Team environment, shared configs | OAuth2 | | Production application | OAuth2 | | Compliance requirements (SOC2, GDPR, etc.) | OAuth2 | ## Choose Your Authentication Method OAuth2 Recommended Best Query Parameter Simplest Authorization Header Programmatic ### OAuth2 Authentication Enterprise-grade security with zero API key exposure. OAuth2 handles authentication automatically through your browser. **Why OAuth2?**- **No key exposure** - API keys never appear in config files or logs - **Token rotation** - Automatic refresh without manual intervention - **Revocable access** - Instantly revoke access without changing keys - **Audit trail** - Track who accessed what and when - **Multi-user ready** - Each user authenticates with their own credentials #### Setup Simply omit the API key from the connection URL: ``` npx mcp-remote https://mcp.scrapfly.io/mcp ``` #### How It Works 1. **Client connects** to MCP server without API key 2. **Server initiates OAuth2 flow** and provides authorization URL 3. **User opens URL in browser** and logs in to Scrapfly 4. **User grants permission** for MCP access 5. **Server receives token** and stores it securely 6. **Future requests** use stored token automatically #### Configuration Examples Claude Desktop Cursor Cline Edit `claude_desktop_config.json`: ``` { "mcpServers": { "scrapfly": { "command": "npx", "args": [ "mcp-remote", "https://mcp.scrapfly.io/mcp" ] } } } ``` **Location:** - **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json` - **Windows:** `%APPDATA%\Claude\claude_desktop_config.json` ``` { "mcpServers": { "scrapfly": { "command": "npx", "args": [ "mcp-remote", "https://mcp.scrapfly.io/mcp" ] } } } ``` ``` { "mcpServers": { "scrapfly": { "command": "npx", "args": [ "mcp-remote", "https://mcp.scrapfly.io/mcp" ] } } } ``` Need more detailed setup instructions? Check our [client-specific integration guides](https://scrapfly.io/docs/mcp/integrations) for step-by-step walkthroughs. #### Token Management **Token Storage:** OAuth2 tokens are stored securely on your local machine: - **macOS:** `~/Library/Application Support/scrapfly-mcp/oauth-token.json` - **Linux:** `~/.config/scrapfly-mcp/oauth-token.json` - **Windows:** `%APPDATA%\scrapfly-mcp\oauth-token.json` **Token Expiration:** Tokens automatically refresh before expiration. If a token expires or is revoked, the MCP server will re-initiate the OAuth2 flow automatically. **Revoke Access:** You can revoke OAuth2 access at any time: 1. Log in to your [Scrapfly Dashboard](https://scrapfly.io/dashboard) 2. Navigate to **Account Settings → API & Integrations** 3. Find "MCP OAuth2 Access" and click **Revoke** #### Best Use Cases - Production deployments - Remote MCP servers accessible by multiple users - Team collaboration environments - Compliance requirements (SOC2, GDPR, etc.) - Any scenario where config files might be shared or committed #### Security Considerations | Security Strengths | Best For | |---|---| | - Zero API key exposure in files - Automatic token rotation - Instant access revocation - Full audit trail - Individual user authentication | - Production deployments - Team collaboration - Remote MCP servers - Compliance environments (SOC2, GDPR) - Shared configuration files | ### Query Parameter Authentication Add your API key directly to the connection URL. Simple, straightforward, and perfect for local development. #### Setup Include the `key` parameter in your MCP connection URL: ``` npx mcp-remote https://mcp.scrapfly.io/mcp?key= ``` #### Configuration Examples Claude Desktop Cursor Cline Edit `claude_desktop_config.json`: ``` { "mcpServers": { "scrapfly": { "command": "npx", "args": [ "mcp-remote", "https://mcp.scrapfly.io/mcp?key=" ] } } } ``` ``` { "mcpServers": { "scrapfly": { "command": "npx", "args": [ "mcp-remote", "https://mcp.scrapfly.io/mcp?key=" ] } } } ``` ``` { "mcpServers": { "scrapfly": { "command": "npx", "args": [ "mcp-remote", "https://mcp.scrapfly.io/mcp?key=" ] } } } ``` #### Project-Scoped Configuration For team collaboration, use environment variables in a project-scoped config: ##### Step 1: Set Environment Variable ``` export SCRAPFLY_API_KEY="" ``` ##### Step 2: Create `.mcp.json` in Project Root ``` { "mcpServers": { "scrapfly": { "command": "npx", "args": [ "mcp-remote", "https://mcp.scrapfly.io/mcp?key=${SCRAPFLY_API_KEY}" ] } } } ``` The `${SCRAPFLY_API_KEY}` syntax automatically uses the environment variable value. Commit `.mcp.json` to version control safely, but add `.env` to `.gitignore`. #### Best Use Cases - Local development and testing on your own machine - Personal projects where security risk is minimal - Quick prototyping and experimentation - Single-user environments with no shared access **Moving to Production?** Consider upgrading to [OAuth2 authentication](#auth-oauth2) for enhanced security and team collaboration features. #### Security Considerations | Security Risks | Mitigation Strategies | |---|---| | - API key visible in config files - May appear in logs - Risk of version control commit - No automatic rotation - Unsuitable for shared environments | - Use `${SCRAPFLY_API_KEY}` env vars - Add `.env` to `.gitignore` - Provide `.env.example` template - Rotate keys manually every 90 days - Use OAuth2 for team/production | **For Local Development Only:** This method is acceptable for personal machines and testing, but always upgrade to OAuth2 before deploying to production or sharing configs with your team. ### Authorization Header Authentication Pass your API key via HTTP Authorization header. Ideal for programmatic access and custom MCP clients. #### Setup Set the `Authorization` header with Bearer token: ``` Authorization: Bearer ``` Connection URL without API key: ``` npx mcp-remote https://mcp.scrapfly.io/mcp ``` #### Python Client Example ``` import asyncio import os from mcp import ClientSession, StdioServerParameters from mcp.client.stdio import stdio_client async def main(): # API key passed via environment or secure storage api_key = os.environ.get("SCRAPFLY_API_KEY") server_params = StdioServerParameters( command="npx", args=["mcp-remote", "https://mcp.scrapfly.io/mcp"], env={ "SCRAPFLY_API_KEY": api_key # Passed to subprocess } ) async with stdio_client(server_params) as (read, write): async with ClientSession(read, write) as session: # Configure session to send Authorization header session.headers["Authorization"] = f"Bearer {api_key}" await session.initialize() tools = await session.list_tools() print(f"Available tools: {[t.name for t in tools]}") asyncio.run(main()) ``` See [MCP Tools & API Specification](https://scrapfly.io/docs/mcp/tools) for a complete list of available tools and their parameters. #### Docker Example Pass API key as environment variable to Docker container: ``` docker run -d \ -e SCRAPFLY_API_KEY="" \ --name mcp-client \ -p 8080:8080 \ my-mcp-client:latest ``` **Tip:** For production deployments, use Docker secrets or Kubernetes secrets instead of `-e` flags to avoid exposing keys in container inspect output. #### Best Use Cases - Server-side applications and backend services - Custom MCP clients (Python, Node.js, etc.) - Automated workflows and CI/CD pipelines - Docker containers and cloud deployments - Environments where query parameters are logged #### Security Considerations | Security Benefits | Security Risks | |---|---| | - Stored in environment variables - Separate from code/config - Headers not logged by default - Better for CI/CD pipelines | - Manual key management - No automatic rotation - Exposed if container compromised - Requires manual revocation | **Best Practice:** Always use environment variables or secret management systems (AWS Secrets Manager, HashiCorp Vault, Kubernetes Secrets) to store API keys. Never hardcode keys in source code. ## Troubleshooting ##### **OAuth2 Authorization Fails** **Problem:** OAuth2 flow doesn't complete or token is rejected. **Solutions:** - Ensure you're logged into the correct Scrapfly account - Check that your account is verified and active - Clear stored tokens and try again: - **macOS:** `rm ~/Library/Application\ Support/scrapfly-mcp/oauth-token.json` - **Linux:** `rm ~/.config/scrapfly-mcp/oauth-token.json` - **Windows:** `del %APPDATA%\scrapfly-mcp\oauth-token.json` - Verify your firewall allows OAuth2 redirects - Check MCP client logs for detailed error messages ##### **API Key Authentication Not Working** **Problem:** Connection fails with API key authentication. **Solutions:** - Verify API key is correct (no extra spaces or line breaks) - Check that the API key belongs to an active project - Ensure the project has remaining credits - Verify the API key hasn't been rotated or revoked in your [dashboard](https://scrapfly.io/dashboard) - Test the API key with a simple curl request to verify it works ##### **Token Refresh Issues** **Problem:** OAuth2 token fails to refresh automatically. **Solutions:** - Delete stored token file (see paths above) and re-authenticate - Check system clock is synchronized (OAuth2 is time-sensitive) - Verify network connectivity to Scrapfly servers - Review MCP client logs for error details - Ensure the refresh token hasn't been revoked in your dashboard ##### **Environment Variables Not Working** **Problem:** `${SCRAPFLY_API_KEY}` syntax doesn't resolve. **Solutions:** - Verify environment variable is set: `echo $SCRAPFLY_API_KEY` - Restart your MCP client after setting the variable - Check that your MCP client supports environment variable expansion - Use absolute path if setting variables in `.env` file ##### Still Having Issues? - Check the [MCP FAQ](https://scrapfly.io/docs/mcp/faq) for common questions and answers - Review your [API key settings](https://scrapfly.io/docs/account) in the dashboard - Visit our [MCP Examples](https://scrapfly.io/docs/mcp/examples) page for working code samples ## Security Best Practices ##### General Guidelines - Never commit API keys to version control - use `.gitignore` - Store keys in environment variables - Rotate keys every 90 days - Use project-specific keys - Monitor usage in [dashboard](https://scrapfly.io/dashboard) ##### OAuth2-Specific - Secure file permissions (`chmod 600`) - Review OAuth2 grants in [dashboard](https://scrapfly.io/dashboard) - Revoke unused authorizations - Use separate dev/prod accounts ##### Team Collaboration - OAuth2 for multi-user teams - Document setup in README - Provide `.env.example` file - Use secret management tools (AWS Secrets Manager, Vault) ## Next Steps [ **Client-Specific Setup Guides** - Claude Desktop, Cursor, Cline, and more ](https://scrapfly.io/docs/mcp/integrations) [ **Getting Started Guide** - Start using MCP with your chosen authentication method ](https://scrapfly.io/docs/mcp/getting-started) [ **Tools & API Specification** - Explore available tools and their capabilities ](https://scrapfly.io/docs/mcp/tools) [ **Manage API Keys** - View and rotate your API keys in the dashboard ](https://scrapfly.io/docs/account) [ **FAQ** - Common questions and answers about MCP ](https://scrapfly.io/docs/mcp/faq)