# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Bring Your Own Proxy](https://scrapfly.io/docs/cloud-browser-api/bring-your-own-proxy) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) - [Native Browser MCP](https://scrapfly.io/docs/cloud-browser-api/mcp) - [DevTools Protocol](https://scrapfly.io/docs/cloud-browser-api/cdp-reference) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Agent Builders [ View as markdown ](https://scrapfly.io/?view=markdown) Copy for LLM Copy for LLM [ Open in ChatGPT ](https://chatgpt.com/?hints=search&prompt=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Fagent-builder%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Claude ](https://claude.ai/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Fagent-builder%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Perplexity ](https://www.perplexity.ai/search/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Fagent-builder%20so%20I%20can%20ask%20questions%20about%20it.) No-code platforms for building AI agents. Integrate Scrapfly web scraping into your visual AI workflows without writing code through API calls and custom tools. No-Code Platforms Relevance AI Flowise LangFlow Dify [ Official Website ](https://relevanceai.com/) 1. [MCP Documentation](https://scrapfly.io/docs/mcp) 2. [Integrations](https://scrapfly.io/docs/mcp/integrations) 3. Agent Builders ## Prerequisites Before getting started, make sure you have the following: - Account on a no-code AI platform (Relevance AI, Flowise, LangFlow, or Dify) - Your Scrapfly API key **Note:** Most no-code platforms do not directly support MCP servers. Instead, we integrate Scrapfly through **HTTP API calls** or **custom tool blocks** in the visual workflow builder. ## Setup Instructions Follow these platform-specific guides to integrate Scrapfly with popular no-code AI agent builders. 1. **Relevance AI Integration** Create a custom tool in Relevance AI that calls Scrapfly's API: 1. Log in to [Relevance AI](https://relevanceai.com/) 2. Navigate to "Tools" → "Custom Tools" 3. Click "Create New Tool" 4. Configure the tool with these settings: **Tool Configuration:** ``` { "name": "Scrape Webpage", "description": "Scrape content from a webpage using Scrapfly", "input_schema": { "url": { "type": "string", "description": "The URL to scrape", "required": true }, "format": { "type": "select", "options": ["markdown", "text", "clean_html"], "default": "markdown" } }, "api_endpoint": "{{ public_api_endpoint }}/scrape", "method": "GET", "params": { "key": "", "url": "{{input.url}}", "format": "{{input.format}}" } } ``` [Sign up for free](https://scrapfly.io/register) to get your API key. Tip: Response MappingConfigure the response mapping to extract content: - **Output Field:** `content` - **JSON Path:** `result.content` 2. **Flowise Integration** Add Scrapfly as a custom tool in your Flowise workflow: 1. Open your Flowise project 2. Drag a "Custom Tool" node onto the canvas 3. Connect it to your LLM node 4. Configure the custom tool: **Tool Code (JavaScript):** ``` const axios = require("axios"); class ScrapflyTool { name = "scrape_webpage"; description = "Scrape content from a webpage using Scrapfly"; async _call(url) { const response = await axios.get("{{ public_api_endpoint }}/scrape", { params: { key: "", url: url, format: "markdown" } }); return response.data.result?.content || "Failed to scrape"; } } module.exports = { nodeClass: ScrapflyTool }; ``` Tip: Error HandlingAdd try-catch blocks to handle API errors gracefully: ``` try { const response = await axios.get(...); return response.data.result?.content; } catch (error) { return `Error: ${error.message}`; } ``` 3. **LangFlow Integration** Use the HTTP Request component to call Scrapfly: 1. Open your LangFlow workflow 2. Add an "HTTP Request" component 3. Configure the component: **HTTP Request Configuration:** - **Method:** GET - **URL:** `https://api.scrapfly.io/scrape` - **Parameters:** - `key`: \_\_API\_KEY\_\_ - `url`: {input\_url} - `format`: markdown - **Output Path:** `result.content` Connect the HTTP Request output to your LLM chain for processing. Tip: Dynamic URLsUse LangFlow variables to pass URLs dynamically from user input or previous nodes. 4. **Dify Integration** Add Scrapfly as an API tool in Dify: 1. Go to [Dify](https://dify.ai/) workspace 2. Navigate to "Tools" → "API Tools" 3. Click "Create API Tool" 4. Fill in the tool details: **API Tool Configuration:** ``` { "name": "Scrapfly Web Scraper", "description": "Scrape web content", "schema": { "url": { "type": "string", "description": "URL to scrape" } }, "api": { "method": "GET", "url": "{{ public_api_endpoint }}/scrape", "params": { "key": "", "url": "{url}", "format": "markdown" } }, "response_mapping": "result.content" } ``` Add the tool to your agent or workflow, and it will be available for the LLM to call. ## Example Workflows ###### Research Agent Scrape this article and summarize the key points ###### Competitive Analysis Compare pricing from these competitor websites ###### News Aggregator Daily digest of top tech news from Hacker News and TechCrunch ###### Product Monitor Monitor this product page and notify me of price changes ## Troubleshooting ##### API Call Fails in No-Code Platform **Problem:** HTTP request to Scrapfly returns errors **Solution:** - Verify API key is correct and active - Check URL parameter is properly formatted (http:// or https://) - Ensure platform allows external API calls - Review platform logs for detailed error messages ##### Response Not Parsing Correctly **Problem:** Cannot extract content from Scrapfly response **Solution:** - Use correct JSON path: `result.content` - Check response structure in API logs - Verify format parameter matches expected output - Test API call outside platform first (Postman, curl) ##### Tool Not Available to Agent **Problem:** Agent does not use the Scrapfly tool **Solution:** - Ensure tool is properly added to agent's available tools - Check tool description is clear and actionable - Use explicit prompts mentioning "scrape" or "fetch data" - Verify tool is enabled in the workflow ##### Rate Limiting Issues **Problem:** Hitting Scrapfly API rate limits **Solution:** - Add delays between requests in workflow - Enable caching in Scrapfly parameters - Upgrade Scrapfly plan for higher limits - Batch similar requests together ##### Timeout Errors **Problem:** Platform times out waiting for Scrapfly response **Solution:** - Increase timeout setting in platform (if available) - Scrape simpler pages or use cache parameter - Disable JavaScript rendering for faster scraping - Contact platform support to adjust timeout limits ## Next Steps - [Explore available MCP tools](https://scrapfly.io/docs/mcp/tools) and their capabilities - [See real-world examples](https://scrapfly.io/docs/mcp/examples) of what you can build - [Learn about authentication methods](https://scrapfly.io/docs/mcp/authentication) in detail - [Read the FAQ](https://scrapfly.io/docs/mcp/faq) for common questions [ Back to All Integrations ](https://scrapfly.io/docs/mcp/integrations)