MCP Authentication

View as markdown

The Scrapfly MCP Server supports three authentication methods, each optimized for different security requirements and deployment scenarios.

New to MCP? Start with our Getting Started Guide to understand the basics before diving into authentication methods.

Quick Comparison

Feature	Query Parameter	Authorization Header	OAuth2
Setup Complexity	Easiest	Moderate	Simple
Security Level	Basic	Good	Best
Key in Config Files	Yes	Via Env Vars	No
Version Control Safe	No	With Env Vars	Yes
Automatic Rotation	No	No	Yes
Instant Revocation	Key Change Required	Key Change Required	Yes
Multi-User Support	Shared Key	Shared Key	Individual Auth
Audit Trail	Limited	Limited	Full

When to Use Each Method

Scenario	Recommended Method
Personal laptop, local development	Query Parameter
Custom Python/Node.js MCP client	Authorization Header
Docker container, cloud deployment	Authorization Header
CI/CD pipeline, automated workflows	Authorization Header
Remote MCP server (accessible by others)	OAuth2
Team environment, shared configs	OAuth2
Production application	OAuth2
Compliance requirements (SOC2, GDPR, etc.)	OAuth2

Choose Your Authentication Method

OAuth2 Authentication

Enterprise-grade security with zero API key exposure. OAuth2 handles authentication automatically through your browser.

Why OAuth2?

No key exposure - API keys never appear in config files or logs
Token rotation - Automatic refresh without manual intervention
Revocable access - Instantly revoke access without changing keys
Audit trail - Track who accessed what and when
Multi-user ready - Each user authenticates with their own credentials

Setup

Simply omit the API key from the connection URL:

npx mcp-remote https://mcp.scrapfly.io/mcp

How It Works

Client connects to MCP server without API key
Server initiates OAuth2 flow and provides authorization URL
User opens URL in browser and logs in to Scrapfly
User grants permission for MCP access
Server receives token and stores it securely
Future requests use stored token automatically

Configuration Examples

Edit claude_desktop_config.json:

{
  "mcpServers": {
    "scrapfly": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://mcp.scrapfly.io/mcp"
      ]
    }
  }
}

Location:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "scrapfly": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://mcp.scrapfly.io/mcp"
      ]
    }
  }
}

{
  "mcpServers": {
    "scrapfly": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://mcp.scrapfly.io/mcp"
      ]
    }
  }
}

Need more detailed setup instructions? Check our client-specific integration guides for step-by-step walkthroughs.

Token Management

Token Storage: OAuth2 tokens are stored securely on your local machine:

macOS: ~/Library/Application Support/scrapfly-mcp/oauth-token.json
Linux: ~/.config/scrapfly-mcp/oauth-token.json
Windows: %APPDATA%\scrapfly-mcp\oauth-token.json

Token Expiration: Tokens automatically refresh before expiration. If a token expires or is revoked, the MCP server will re-initiate the OAuth2 flow automatically.

Revoke Access: You can revoke OAuth2 access at any time:

Log in to your Scrapfly Dashboard
Navigate to Account Settings → API & Integrations
Find "MCP OAuth2 Access" and click Revoke

Best Use Cases

Production deployments
Remote MCP servers accessible by multiple users
Team collaboration environments
Compliance requirements (SOC2, GDPR, etc.)
Any scenario where config files might be shared or committed

Security Considerations

Security Strengths	Best For
Zero API key exposure in files Automatic token rotation Instant access revocation Full audit trail Individual user authentication	Production deployments Team collaboration Remote MCP servers Compliance environments (SOC2, GDPR) Shared configuration files

Query Parameter Authentication

Add your API key directly to the connection URL. Simple, straightforward, and perfect for local development.

Setup

Include the key parameter in your MCP connection URL:

npx mcp-remote https://mcp.scrapfly.io/mcp?key=

Configuration Examples

Edit claude_desktop_config.json:

{
  "mcpServers": {
    "scrapfly": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://mcp.scrapfly.io/mcp?key="
      ]
    }
  }
}

{
  "mcpServers": {
    "scrapfly": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://mcp.scrapfly.io/mcp?key="
      ]
    }
  }
}

{
  "mcpServers": {
    "scrapfly": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://mcp.scrapfly.io/mcp?key="
      ]
    }
  }
}

Project-Scoped Configuration

For team collaboration, use environment variables in a project-scoped config:

Step 1: Set Environment Variable

export SCRAPFLY_API_KEY=""

Step 2: Create `.mcp.json` in Project Root

{
  "mcpServers": {
    "scrapfly": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://mcp.scrapfly.io/mcp?key=${SCRAPFLY_API_KEY}"
      ]
    }
  }
}

The ${SCRAPFLY_API_KEY} syntax automatically uses the environment variable value. Commit .mcp.json to version control safely, but add .env to .gitignore.

Best Use Cases

Local development and testing on your own machine
Personal projects where security risk is minimal
Quick prototyping and experimentation
Single-user environments with no shared access

Moving to Production? Consider upgrading to OAuth2 authentication for enhanced security and team collaboration features.

Security Considerations

Security Risks	Mitigation Strategies
API key visible in config files May appear in logs Risk of version control commit No automatic rotation Unsuitable for shared environments	Use `${SCRAPFLY_API_KEY}` env vars Add `.env` to `.gitignore` Provide `.env.example` template Rotate keys manually every 90 days Use OAuth2 for team/production

For Local Development Only: This method is acceptable for personal machines and testing, but always upgrade to OAuth2 before deploying to production or sharing configs with your team.

Authorization Header Authentication

Pass your API key via HTTP Authorization header. Ideal for programmatic access and custom MCP clients.

Setup

Set the Authorization header with Bearer token:

Authorization: Bearer

Connection URL without API key:

npx mcp-remote https://mcp.scrapfly.io/mcp

Python Client Example

import asyncio
import os
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def main():
    # API key passed via environment or secure storage
    api_key = os.environ.get("SCRAPFLY_API_KEY")

    server_params = StdioServerParameters(
        command="npx",
        args=["mcp-remote", "https://mcp.scrapfly.io/mcp"],
        env={
            "SCRAPFLY_API_KEY": api_key  # Passed to subprocess
        }
    )

    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            # Configure session to send Authorization header
            session.headers["Authorization"] = f"Bearer {api_key}"

            await session.initialize()
            tools = await session.list_tools()
            print(f"Available tools: {[t.name for t in tools]}")

asyncio.run(main())

See MCP Tools & API Specification for a complete list of available tools and their parameters.

Docker Example

Pass API key as environment variable to Docker container:

docker run -d \
  -e SCRAPFLY_API_KEY="" \
  --name mcp-client \
  -p 8080:8080 \
  my-mcp-client:latest

Tip: For production deployments, use Docker secrets or Kubernetes secrets instead of -e flags to avoid exposing keys in container inspect output.

Best Use Cases

Server-side applications and backend services
Custom MCP clients (Python, Node.js, etc.)
Automated workflows and CI/CD pipelines
Docker containers and cloud deployments
Environments where query parameters are logged

Security Considerations

Security Benefits	Security Risks
Stored in environment variables Separate from code/config Headers not logged by default Better for CI/CD pipelines	Manual key management No automatic rotation Exposed if container compromised Requires manual revocation

Best Practice: Always use environment variables or secret management systems (AWS Secrets Manager, HashiCorp Vault, Kubernetes Secrets) to store API keys. Never hardcode keys in source code.

Troubleshooting

Problem: OAuth2 flow doesn't complete or token is rejected.

Solutions:

Ensure you're logged into the correct Scrapfly account
Check that your account is verified and active
Clear stored tokens and try again:
- macOS: rm ~/Library/Application\ Support/scrapfly-mcp/oauth-token.json
- Linux: rm ~/.config/scrapfly-mcp/oauth-token.json
- Windows: del %APPDATA%\scrapfly-mcp\oauth-token.json
Verify your firewall allows OAuth2 redirects
Check MCP client logs for detailed error messages

Problem: Connection fails with API key authentication.

Solutions:

Verify API key is correct (no extra spaces or line breaks)
Check that the API key belongs to an active project
Ensure the project has remaining credits
Verify the API key hasn't been rotated or revoked in your dashboard
Test the API key with a simple curl request to verify it works

Problem: OAuth2 token fails to refresh automatically.

Solutions:

Delete stored token file (see paths above) and re-authenticate
Check system clock is synchronized (OAuth2 is time-sensitive)
Verify network connectivity to Scrapfly servers
Review MCP client logs for error details
Ensure the refresh token hasn't been revoked in your dashboard

Problem: ${SCRAPFLY_API_KEY} syntax doesn't resolve.

Solutions:

Verify environment variable is set: echo $SCRAPFLY_API_KEY
Restart your MCP client after setting the variable
Check that your MCP client supports environment variable expansion
Use absolute path if setting variables in .env file

Still Having Issues?

Check the MCP FAQ for common questions and answers
Review your API key settings in the dashboard
Visit our MCP Examples page for working code samples

Security Best Practices

General Guidelines

Never commit API keys to version control - use .gitignore
Store keys in environment variables
Rotate keys every 90 days
Use project-specific keys
Monitor usage in dashboard

OAuth2-Specific

Secure file permissions (chmod 600)
Review OAuth2 grants in dashboard
Revoke unused authorizations
Use separate dev/prod accounts

Team Collaboration

OAuth2 for multi-user teams
Document setup in README
Provide .env.example file
Use secret management tools (AWS Secrets Manager, Vault)

Next Steps

Client-Specific Setup Guides - Claude Desktop, Cursor, Cline, and more Getting Started Guide - Start using MCP with your chosen authentication method Tools & API Specification - Explore available tools and their capabilities Manage API Keys - View and rotate your API keys in the dashboard FAQ - Common questions and answers about MCP

MCP Authentication

Quick Comparison

When to Use Each Method

Choose Your Authentication Method

OAuth2 Authentication

Setup

How It Works

Configuration Examples

Token Management

Best Use Cases

Security Considerations

Query Parameter Authentication

Setup

Configuration Examples

Project-Scoped Configuration

Step 1: Set Environment Variable

Step 2: Create .mcp.json in Project Root

Best Use Cases

Security Considerations

Authorization Header Authentication

Setup

Python Client Example

Docker Example

Best Use Cases

Security Considerations

Troubleshooting

OAuth2 Authorization Fails

API Key Authentication Not Working

Token Refresh Issues

Environment Variables Not Working

Still Having Issues?

Security Best Practices

General Guidelines

OAuth2-Specific

Team Collaboration

Next Steps

Summary

Step 2: Create `.mcp.json` in Project Root