# Scrapfly Documentation

## Table of Contents

### Dashboard

- [Intro](https://scrapfly.io/docs)
- [Project](https://scrapfly.io/docs/project)
- [Account](https://scrapfly.io/docs/account)
- [Workspace & Team](https://scrapfly.io/docs/workspace-and-team)
- [Billing](https://scrapfly.io/docs/billing)

### Products

#### MCP Server

- [Getting Started](https://scrapfly.io/docs/mcp/getting-started)
- [Tools & API Spec](https://scrapfly.io/docs/mcp/tools)
- [Authentication](https://scrapfly.io/docs/mcp/authentication)
- [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples)
- [FAQ](https://scrapfly.io/docs/mcp/faq)
##### Integrations

- [Overview](https://scrapfly.io/docs/mcp/integrations)
- [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop)
- [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code)
- [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt)
- [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor)
- [Cline](https://scrapfly.io/docs/mcp/integrations/cline)
- [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf)
- [Zed](https://scrapfly.io/docs/mcp/integrations/zed)
- [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code)
- [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode)
- [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain)
- [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex)
- [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai)
- [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai)
- [n8n](https://scrapfly.io/docs/mcp/integrations/n8n)
- [Make](https://scrapfly.io/docs/mcp/integrations/make)
- [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier)
- [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi)
- [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder)
- [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client)


#### Web Scraping API

- [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started)
- [API Specification](https://scrapfly.io/docs/scrape-api/specification)
- [Monitoring](https://scrapfly.io/docs/monitoring)
- [Customize Request](https://scrapfly.io/docs/scrape-api/custom)
- [Debug](https://scrapfly.io/docs/scrape-api/debug)
- [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection)
- [Proxy](https://scrapfly.io/docs/scrape-api/proxy)
- [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode)
- [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog)
- [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify)
- [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction)
- [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering)
- [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario)
- [SSL](https://scrapfly.io/docs/scrape-api/ssl)
- [DNS](https://scrapfly.io/docs/scrape-api/dns)
- [Cache](https://scrapfly.io/docs/scrape-api/cache)
- [Session](https://scrapfly.io/docs/scrape-api/session)
- [Webhook](https://scrapfly.io/docs/scrape-api/webhook)
- [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot)
- [Errors](https://scrapfly.io/docs/scrape-api/errors)
- [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout)
- [Throttling](https://scrapfly.io/docs/throttling)
- [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot)
- [Billing](https://scrapfly.io/docs/scrape-api/billing)
- [FAQ](https://scrapfly.io/docs/scrape-api/faq)

#### Crawler API

- [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started)
- [API Specification](https://scrapfly.io/docs/crawler-api/specification)
- [Retrieving Results](https://scrapfly.io/docs/crawler-api/results)
- [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format)
- [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules)
- [Webhook](https://scrapfly.io/docs/crawler-api/webhook)
- [Billing](https://scrapfly.io/docs/crawler-api/billing)
- [Errors](https://scrapfly.io/docs/crawler-api/errors)
- [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot)
- [FAQ](https://scrapfly.io/docs/crawler-api/faq)

#### Screenshot API

- [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started)
- [API Specification](https://scrapfly.io/docs/screenshot-api/specification)
- [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility)
- [Webhook](https://scrapfly.io/docs/screenshot-api/webhook)
- [Billing](https://scrapfly.io/docs/screenshot-api/billing)
- [Errors](https://scrapfly.io/docs/screenshot-api/errors)

#### Extraction API

- [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started)
- [API Specification](https://scrapfly.io/docs/extraction-api/specification)
- [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template)
- [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt)
- [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai)
- [Webhook](https://scrapfly.io/docs/extraction-api/webhook)
- [Billing](https://scrapfly.io/docs/extraction-api/billing)
- [Errors](https://scrapfly.io/docs/extraction-api/errors)
- [FAQ](https://scrapfly.io/docs/extraction-api/faq)

#### Proxy Saver

- [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started)
- [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints)
- [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations)
- [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates)
- [Protocols](https://scrapfly.io/docs/proxy-saver/protocols)
- [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile)
- [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security)
- [Billing](https://scrapfly.io/docs/proxy-saver/billing)

#### Cloud Browser API

- [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started)
- [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy)
- [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock)
- [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads)
- [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume)
- [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop)
- [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode)
- [Bring Your Own Proxy](https://scrapfly.io/docs/cloud-browser-api/bring-your-own-proxy)
- [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions)
##### Integrations

- [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer)
- [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright)
- [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium)
- [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser)
- [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use)
- [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand)
- [Vibium](https://scrapfly.io/docs/cloud-browser-api/vibium)

- [Billing](https://scrapfly.io/docs/cloud-browser-api/billing)
- [Errors](https://scrapfly.io/docs/cloud-browser-api/errors)


### Tools

- [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector)

### SDK

- [Golang](https://scrapfly.io/docs/sdk/golang)
- [Python](https://scrapfly.io/docs/sdk/python)
- [TypeScript](https://scrapfly.io/docs/sdk/typescript)
- [Scrapy](https://scrapfly.io/docs/sdk/scrapy)

### Integrations

- [Getting Started](https://scrapfly.io/docs/integration/getting-started)
- [LangChain](https://scrapfly.io/docs/integration/langchain)
- [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex)
- [CrewAI](https://scrapfly.io/docs/integration/crewai)
- [Zapier](https://scrapfly.io/docs/integration/zapier)
- [Make](https://scrapfly.io/docs/integration/make)
- [n8n](https://scrapfly.io/docs/integration/n8n)

### Academy

- [Overview](https://scrapfly.io/academy)
- [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview)
- [Tools](https://scrapfly.io/academy/tools-overview)
- [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering)
- [Static Scraping](https://scrapfly.io/academy/static-scraping)
- [HTML Parsing](https://scrapfly.io/academy/html-parsing)
- [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping)
- [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping)
- [Headless Browsers](https://scrapfly.io/academy/headless-browsers)
- [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data)
- [JSON Parsing](https://scrapfly.io/academy/json-parsing)
- [Data Processing](https://scrapfly.io/academy/data-processing)
- [Scaling](https://scrapfly.io/academy/scaling)
- [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary)
- [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking)
- [Proxies](https://scrapfly.io/academy/proxies)

---

# LangChain

 [  View as markdown ](https://scrapfly.io/?view=markdown)   Copy for LLM    Copy for LLM  [     Open in ChatGPT ](https://chatgpt.com/?hints=search&prompt=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Flangchain%20so%20I%20can%20ask%20questions%20about%20it.) [     Open in Claude ](https://claude.ai/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Flangchain%20so%20I%20can%20ask%20questions%20about%20it.) [     Open in Perplexity ](https://www.perplexity.ai/search/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Flangchain%20so%20I%20can%20ask%20questions%20about%20it.) 

 

 

 Popular AI framework for building LLM applications. Integrate Scrapfly web scraping into your LangChain agents and chains for intelligent data collection workflows.

 

 

 AI Framework Python JavaScript TypeScript [  Official Website ](https://python.langchain.com/docs/integrations/tools/mcp) 

 

 1. [MCP Documentation](https://scrapfly.io/docs/mcp)
2. [Integrations](https://scrapfly.io/docs/mcp/integrations)
3. LangChain
 
 ## Prerequisites

Before getting started, make sure you have the following:

- Python 3.8+ or Node.js 18+ installed
- `langchain-mcp-adapters` or `@langchain/mcp-adapters` packages installed
- Your Scrapfly API key
 
## Setup Instructions

LangChain supports MCP servers through the `langchain-mcp` integration. Follow these steps to connect Scrapfly.

1. **Install Required Packages** Install LangChain and the MCP integration package:
    
    **Python:**
    
     ```
    pip install langchain-mcp-adapters
    ```
    
     
    
       
    
     
    
    **JavaScript/TypeScript:**
    
     ```
    npm install @langchain/mcp-adapters
    ```
    
     
    
       
    
     
    
      Tip: Virtual EnvironmentsFor Python, use a virtual environment to avoid package conflicts:
    
     ```
    python -m venv venv
    source venv/bin/activate  # On Windows: venv\Scripts\activate
    pip install langchain-mcp-adapters
    ```
2. **Initialize MCP Client in Your Code** Connect to the Scrapfly MCP server from your LangChain application:
    
      **Team collaboration:** [See project-scoped setup](https://scrapfly.io/docs/mcp/authentication#project-scoped) to share configuration with your team via version control.
    
    ### Python Example
    
     ```
    import asyncio
    
    from langchain_mcp_adapters.client import MultiServerMCPClient
    
    # Initialize Scrapfly MCP client with OAuth2
    mcp_client = MultiServerMCPClient(
        {
            "scrapfly-mcp": {
                "transport": "http",
                "url": "https://mcp.scrapfly.io/mcp?key=YOUR_API_KEY",
            }
        }
    )
    
    async def main():
        # Get available tools
        tools = await mcp_client.get_tools()
    
        print(f"Loaded {len(tools)} Scrapfly tools")
        for tool in tools:
            print(f"  - {tool.name}")
    
    asyncio.run(main())
    ```
    
     
    
       
    
     
    
      **Important:** Replace `YOUR_API_KEY` with your actual Scrapfly API key. [Sign up for free](https://scrapfly.io/register) to get your API key. 
    
    ### JavaScript/TypeScript Example
    
     ```
    import { MultiServerMCPClient } from "@langchain/mcp-adapters";
    
    const client = new MultiServerMCPClient({
        scrapflyMcp: {
            transport: "http",
            url: "https://mcp.scrapfly.io/mcp?key=YOUR_API_KEY",
        },
    });
    
    const tools = await client.getTools();
    for (const tool of tools) {
        console.log(`- ${tool.name}`);
    }
    ```
3. **Build a LangChain Agent with Scrapfly Tools** Create an AI agent that can use Scrapfly for web scraping:
    
     ```
    from langchain_anthropic import ChatAnthropic
    from langchain.agents import AgentExecutor, create_tool_calling_agent
    from langchain.prompts import ChatPromptTemplate
    from langchain_mcp_adapters.client import MultiServerMCPClient
    
    # Initialize Scrapfly MCP
    mcp_client = MultiServerMCPClient(
        {
            "scrapfly-mcp": {
                "transport": "http",
                "url": "https://mcp.scrapfly.io/mcp?key=YOUR_API_KEY",
            }
        }
    )
    
    # Get Scrapfly tools
    scrapfly_tools = mcp_client.get_tools()
    
    # Initialize LLM
    llm = ChatAnthropic(model="claude-3-5-sonnet-20241022")
    
    # Create prompt template
    prompt = ChatPromptTemplate.from_messages([
        ("system", "You are a helpful AI assistant with web scraping capabilities."),
        ("human", "{input}"),
        ("placeholder", "{agent_scratchpad}")
    ])
    
    # Create agent with Scrapfly tools
    agent = create_tool_calling_agent(llm, scrapfly_tools, prompt)
    agent_executor = AgentExecutor(agent=agent, tools=scrapfly_tools, verbose=True)
    
    # Use the agent
    result = agent_executor.invoke({
        "input": "Scrape the top 5 posts from Hacker News and summarize them"
    })
    
    print(result["output"])
    ```
    
     
    
       
    
     
    
      **Pro Tip:** The agent will automatically call `scraping_instruction_enhanced` to get required parameters before scraping!
4. **Test Your Integration** Run your LangChain application and verify Scrapfly MCP is working:
    
     ```
    python your_langchain_app.py
    ```
    
     
    
       
    
     
    
      Tip: Enable Verbose ModeSet `verbose=True` in `AgentExecutor` to see detailed logs of tool calls and agent reasoning.

## Example Prompts

###### Research Assistant Agent

    

Research the top AI news from today and create a summary report with sources

 

    

###### Competitive Analysis Chain

    

Scrape pricing data from competitor websites and create a comparison table

 

    

###### Data Pipeline with LLM Processing

    

Scrape product reviews, analyze sentiment, and generate insights

 

    

###### Multi-Step Research Workflow

    

Find the top 3 blog posts about web scraping, scrape their content, and summarize key takeaways

 

    



## Troubleshooting

#####    Import Error: langchain-mcp not found   

 

**Problem:** `ModuleNotFoundError: No module named 'langchain_mcp'`

**Solution:**

- Ensure you installed the package: `pip install langchain-mcp`
- Verify you're using the correct Python environment (check with `which python`)
- Try upgrading: `pip install --upgrade langchain-mcp`
 
 

 

 

#####    npx Command Not Found   

 

**Problem:** MCPClient cannot execute `npx` command

**Solution:**

- Ensure Node.js 18+ is installed: `node --version`
- Verify `npx` is in PATH: `npx --version`
- On Windows, restart terminal after installing Node.js
- Try specifying full path: `command="/usr/local/bin/npx"`
 
 

 

 

#####    OAuth2 Authorization in Headless Environment   

 

**Problem:** OAuth2 link cannot open in server/CI environment

**Solution:**

- Use API key authentication instead of OAuth2
- Set API key as environment variable: `SCRAPFLY_API_KEY`
- Load from environment: `args=["mcp-remote", f"https://mcp.scrapfly.io/mcp?key={os.getenv('SCRAPFLY_API_KEY')}"]`
 
 

 

 

#####    Agent Not Calling Scrapfly Tools   

 

**Problem:** Agent does not use Scrapfly tools even when prompted

**Solution:**

- Verify tools are loaded: `print([tool.name for tool in scrapfly_tools])`
- Check LLM supports tool calling (Claude 3+, GPT-4+)
- Use more explicit prompts mentioning "scrape" or "web data"
- Enable verbose mode to see agent's reasoning
 
 

 

 

#####    MCP Connection Timeout   

 

**Problem:** Connection to Scrapfly MCP server times out

**Solution:**

- Check internet connection
- Verify `https://mcp.scrapfly.io/mcp` is accessible
- If behind a proxy, configure proxy settings
- Increase timeout in MCPClient initialization if needed
 
 

 

 



## Next Steps

- [Explore available MCP tools](https://scrapfly.io/docs/mcp/tools) and their capabilities
- [See real-world examples](https://scrapfly.io/docs/mcp/examples) of what you can build
- [Learn about authentication methods](https://scrapfly.io/docs/mcp/authentication) in detail
- [Read the FAQ](https://scrapfly.io/docs/mcp/faq) for common questions
 
 [  Back to All Integrations ](https://scrapfly.io/docs/mcp/integrations)