Prerequisites

Before getting started, make sure you have the following:

• Python 3.8+ or Node.js 18+ installed
• langchain and langchain-mcp packages installed
• Your Scrapfly API key (only if not using OAuth2)

Setup Instructions

LangChain supports MCP servers through the langchain-mcp integration. Follow these steps to connect Scrapfly.

1. Install Required Packages

   Install LangChain and the MCP integration package:

   Python:

   pip install langchain langchain-mcp langchain-anthropic

   JavaScript/TypeScript:

   npm install langchain @langchain/anthropic @modelcontextprotocol/sdk
   Tip: Virtual Environments

   For Python, use a virtual environment to avoid package conflicts:

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
   pip install langchain langchain-mcp langchain-anthropic
2. Initialize MCP Client in Your Code

   Connect to the Scrapfly MCP server from your LangChain application:

   Python Example with OAuth2 (Recommended)

   from langchain_mcp import MCPClient
   
   # Initialize Scrapfly MCP client with OAuth2
   mcp_client = MCPClient(
       name="scrapfly",
       command="npx",
       args=["mcp-remote", "https://mcp.scrapfly.io/mcp"]
   )
   
   # Get available tools
   tools = mcp_client.get_tools()
   
   print(f"Loaded {len(tools)} Scrapfly tools")
   for tool in tools:
       print(f"  - {tool.name}")
   Why OAuth2?

   • No API keys in your code repository
   • Automatic token rotation for enhanced security
   • Instant revocation if needed
   • Full audit trail of authentication events

   Team collaboration: See project-scoped setup to share configuration with your team via version control.

   Python Example with API Key

   from langchain_mcp import MCPClient
   
   # Initialize Scrapfly MCP client with API key
   mcp_client = MCPClient(
       name="scrapfly",
       command="npx",
       args=["mcp-remote", "https://mcp.scrapfly.io/mcp?key=YOUR_API_KEY"]
   )
   
   tools = mcp_client.get_tools()
   Important: Replace YOUR_API_KEY with your actual Scrapfly API key. Sign up for free to get your API key.

   JavaScript/TypeScript Example

   import { MCPClient } from "@modelcontextprotocol/sdk";
   
   // Initialize Scrapfly MCP client
   const mcpClient = new MCPClient({
     name: "scrapfly",
     command: "npx",
     args: ["mcp-remote", "https://mcp.scrapfly.io/mcp"]
   });
   
   // Get available tools
   const tools = await mcpClient.getTools();
   console.log(`Loaded ${tools.length} Scrapfly tools`);
3. Build a LangChain Agent with Scrapfly Tools

   Create an AI agent that can use Scrapfly for web scraping:

   from langchain_anthropic import ChatAnthropic
   from langchain.agents import AgentExecutor, create_tool_calling_agent
   from langchain.prompts import ChatPromptTemplate
   from langchain_mcp import MCPClient
   
   # Initialize Scrapfly MCP
   mcp_client = MCPClient(
       name="scrapfly",
       command="npx",
       args=["mcp-remote", "https://mcp.scrapfly.io/mcp"]
   )
   
   # Get Scrapfly tools
   scrapfly_tools = mcp_client.get_tools()
   
   # Initialize LLM
   llm = ChatAnthropic(model="claude-3-5-sonnet-20241022")
   
   # Create prompt template
   prompt = ChatPromptTemplate.from_messages([
       ("system", "You are a helpful AI assistant with web scraping capabilities."),
       ("human", "{input}"),
       ("placeholder", "{agent_scratchpad}")
   ])
   
   # Create agent with Scrapfly tools
   agent = create_tool_calling_agent(llm, scrapfly_tools, prompt)
   agent_executor = AgentExecutor(agent=agent, tools=scrapfly_tools, verbose=True)
   
   # Use the agent
   result = agent_executor.invoke({
       "input": "Scrape the top 5 posts from Hacker News and summarize them"
   })
   
   print(result["output"])
   Pro Tip: The agent will automatically call scraping_instruction_enhanced to get required parameters before scraping!
4. Test Your Integration

   Run your LangChain application and verify Scrapfly MCP is working:

   python your_langchain_app.py
   Tip: Enable Verbose Mode

   Set verbose=True in AgentExecutor to see detailed logs of tool calls and agent reasoning.

Example Prompts

Troubleshooting

Next Steps

• Explore available MCP tools and their capabilities
• See real-world examples of what you can build
• Learn about authentication methods in detail
• Read the FAQ for common questions