# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Bring Your Own Proxy](https://scrapfly.io/docs/cloud-browser-api/bring-your-own-proxy) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) - [Native Browser MCP](https://scrapfly.io/docs/cloud-browser-api/mcp) - [DevTools Protocol](https://scrapfly.io/docs/cloud-browser-api/cdp-reference) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Custom MCP Client [ View as markdown ](https://scrapfly.io/?view=markdown) Copy for LLM Copy for LLM [ Open in ChatGPT ](https://chatgpt.com/?hints=search&prompt=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Fcustom-client%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Claude ](https://claude.ai/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Fcustom-client%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Perplexity ](https://www.perplexity.ai/search/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Fcustom-client%20so%20I%20can%20ask%20questions%20about%20it.) Build your own MCP client to integrate Scrapfly into custom applications. Complete guide for developers creating native MCP integrations with the Model Context Protocol SDK. Developer Guide Python JavaScript TypeScript Any Language [ Official Website ](https://modelcontextprotocol.io/introduction) 1. [MCP Documentation](https://scrapfly.io/docs/mcp) 2. [Integrations](https://scrapfly.io/docs/mcp/integrations) 3. Custom MCP Client ## Prerequisites Before getting started, make sure you have the following: - Programming experience in Python, JavaScript, or TypeScript - Understanding of async/await and HTTP protocols - Node.js 18+ or Python 3.8+ installed - Your Scrapfly API key (only if not using OAuth2) ## Overview The Model Context Protocol (MCP) is a standard for connecting AI applications to external data sources and tools. This guide shows you how to build a custom MCP client that connects to the Scrapfly MCP server. **What You'll Build:** An MCP client application that connects to `https://mcp.scrapfly.io/mcp` and can call Scrapfly's web scraping tools programmatically. ## Setup Instructions 1. **Install MCP SDK** Install the official Model Context Protocol SDK for your language: **Python:** ``` pip install mcp anthropic ``` **JavaScript/TypeScript:** ``` npm install @modelcontextprotocol/sdk @anthropic-ai/sdk ``` Tip: MCP Protocol SpecificationFor advanced use cases, review the official MCP specification: 2. **Create Basic MCP Client (Python)** Build a simple MCP client that connects to Scrapfly MCP server: ``` import asyncio from mcp import ClientSession, StdioServerParameters from mcp.client.stdio import stdio_client from anthropic import Anthropic async def main(): # Define Scrapfly MCP server connection server_params = StdioServerParameters( command="npx", args=["mcp-remote", "https://mcp.scrapfly.io/mcp"], env=None ) # Connect to Scrapfly MCP server async with stdio_client(server_params) as (read, write): async with ClientSession(read, write) as session: # Initialize the session await session.initialize() # List available tools tools = await session.list_tools() print(f"Available Scrapfly tools: {len(tools.tools)}") for tool in tools.tools: print(f" - {tool.name}: {tool.description}") # Example: Call web_get_page tool result = await session.call_tool( name="web_get_page", arguments={ "url": "https://news.ycombinator.com", "pow": "..." # Get this from scraping_instruction_enhanced first } ) print("Scraped content:") print(result.content[0].text) # Run the client asyncio.run(main()) ``` **Important:** The `pow` parameter is required for scraping. Call `scraping_instruction_enhanced` first to get it, or use OAuth2 which handles this automatically. 3. **Create Basic MCP Client (JavaScript/TypeScript)** Build an MCP client in JavaScript or TypeScript: ``` import { Client } from "@modelcontextprotocol/sdk/client/index.js"; import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js"; async function main() { // Create transport to Scrapfly MCP server const transport = new StdioClientTransport({ command: "npx", args: ["mcp-remote", "https://mcp.scrapfly.io/mcp"] }); // Create MCP client const client = new Client( { name: "scrapfly-client", version: "1.0.0" }, { capabilities: {} } ); // Connect to server await client.connect(transport); // List available tools const tools = await client.listTools(); console.log(`Available Scrapfly tools: ${tools.tools.length}`); tools.tools.forEach(tool => { console.log(` - ${tool.name}: ${tool.description}`); }); // Call web_get_page tool const result = await client.callTool({ name: "web_get_page", arguments: { url: "https://news.ycombinator.com", pow: "..." // Get from scraping_instruction_enhanced } }); console.log("Scraped content:"); console.log(result.content[0].text); // Close connection await client.close(); } main().catch(console.error); ``` 4. **Implement OAuth2 Authentication** For production applications, use OAuth2 instead of API keys: ### Python OAuth2 Flow ``` import asyncio import webbrowser from mcp import ClientSession, StdioServerParameters from mcp.client.stdio import stdio_client async def connect_with_oauth(): """Connect to Scrapfly MCP with OAuth2""" server_params = StdioServerParameters( command="npx", args=["mcp-remote", "https://mcp.scrapfly.io/mcp"] # No API key in args - OAuth2 will handle authentication ) async with stdio_client(server_params) as (read, write): async with ClientSession(read, write) as session: await session.initialize() # On first connection, the server will return an OAuth2 URL # You can detect this and open the browser automatically: # webbrowser.open(oauth_url) # After OAuth2 authorization, tools become available tools = await session.list_tools() print(f"Authenticated! {len(tools.tools)} tools available") return session asyncio.run(connect_with_oauth()) ``` **OAuth2 Benefits:** - No API keys in code or config files - Automatic token refresh - Better security and audit trail - User-specific authentication 5. **Build AI Agent with MCP Tools** Integrate Scrapfly MCP tools with an AI agent using Anthropic SDK: ``` import asyncio from mcp import ClientSession, StdioServerParameters from mcp.client.stdio import stdio_client from anthropic import Anthropic async def build_agent(): """Build an AI agent that can use Scrapfly tools""" # Connect to Scrapfly MCP server_params = StdioServerParameters( command="npx", args=["mcp-remote", "https://mcp.scrapfly.io/mcp"] ) async with stdio_client(server_params) as (read, write): async with ClientSession(read, write) as session: await session.initialize() # Get available tools tools_response = await session.list_tools() tools = tools_response.tools # Initialize Anthropic client client = Anthropic() # Create AI agent with Scrapfly tools messages = [{ "role": "user", "content": "Scrape the top 5 posts from Hacker News and summarize them" }] # Agent loop while True: response = client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=4096, tools=[{ "name": tool.name, "description": tool.description, "input_schema": tool.inputSchema } for tool in tools], messages=messages ) # Check if agent wants to use a tool if response.stop_reason == "tool_use": tool_use = next( block for block in response.content if block.type == "tool_use" ) # Call the tool via MCP tool_result = await session.call_tool( name=tool_use.name, arguments=tool_use.input ) # Add tool result to conversation messages.append({ "role": "assistant", "content": response.content }) messages.append({ "role": "user", "content": [{ "type": "tool_result", "tool_use_id": tool_use.id, "content": tool_result.content[0].text }] }) else: # Agent is done print(response.content[0].text) break asyncio.run(build_agent()) ``` **Pro Tip:** This agent will automatically call `scraping_instruction_enhanced` first to get required parameters! 6. **Error Handling and Best Practices** Implement robust error handling for production applications: ``` import asyncio from mcp import ClientSession, StdioServerParameters from mcp.client.stdio import stdio_client async def robust_mcp_client(): """MCP client with error handling""" try: server_params = StdioServerParameters( command="npx", args=["mcp-remote", "https://mcp.scrapfly.io/mcp"] ) async with stdio_client(server_params) as (read, write): async with ClientSession(read, write) as session: # Initialize with timeout await asyncio.wait_for( session.initialize(), timeout=30.0 ) # Call tool with error handling try: result = await session.call_tool( name="web_get_page", arguments={ "url": "https://web-scraping.dev", "pow": "..." } ) if result.isError: print(f"Tool error: {result.content[0].text}") else: print(f"Success: {result.content[0].text}") except Exception as tool_error: print(f"Tool call failed: {tool_error}") except asyncio.TimeoutError: print("Connection timeout - MCP server not responding") except Exception as e: print(f"MCP client error: {e}") asyncio.run(robust_mcp_client()) ``` ## Example Prompts ###### Custom Research Assistant Build a command-line tool that scrapes web pages and answers questions using Claude ###### Internal Automation Platform Create an internal tool that integrates web scraping into your company's workflows ###### Data Collection Pipeline Build a scheduled job that scrapes data and stores it in your database ###### Custom AI Agent Develop a specialized AI agent with web scraping capabilities for your use case ## Troubleshooting ##### MCP Server Connection Fails **Problem:** Cannot connect to Scrapfly MCP server **Solution:** - Verify `npx` is available: `npx --version` - Check internet connectivity to `https://mcp.scrapfly.io/mcp` - Ensure Node.js 18+ is installed - Try running `npx mcp-remote https://mcp.scrapfly.io/mcp` manually ##### Tool Calls Failing **Problem:** Tool calls return errors or fail **Solution:** - Ensure you call `scraping_instruction_enhanced` first to get `pow` parameter - Check tool arguments match expected schema - Verify API key or OAuth2 authentication is working - Review error messages in tool call results ##### OAuth2 Flow Not Working **Problem:** OAuth2 authorization does not complete **Solution:** - Detect OAuth2 URL in initial connection response - Open URL in browser automatically or prompt user - Wait for OAuth2 completion before making tool calls - For headless environments, use API key authentication instead ##### Session Timeout Issues **Problem:** MCP session times out or disconnects **Solution:** - Implement connection retry logic - Keep session alive with periodic health checks - Reconnect automatically on disconnect - Increase timeout values for long-running operations ##### Type/Schema Errors **Problem:** Tool schema or argument type errors **Solution:** - Use `list_tools()` to get exact schema for each tool - Validate arguments against schema before calling - Check parameter types (string, boolean, integer, etc.) - Review MCP protocol specification for correct formats ##### Performance Issues **Problem:** Slow tool calls or high latency **Solution:** - Implement caching for frequently scraped URLs - Use async/await properly to avoid blocking - Batch similar requests when possible - Monitor network latency and optimize connection settings ## Next Steps - [Explore available MCP tools](https://scrapfly.io/docs/mcp/tools) and their capabilities - [See real-world examples](https://scrapfly.io/docs/mcp/examples) of what you can build - [Learn about authentication methods](https://scrapfly.io/docs/mcp/authentication) in detail - [Read the FAQ](https://scrapfly.io/docs/mcp/faq) for common questions [ Back to All Integrations ](https://scrapfly.io/docs/mcp/integrations)