# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Bring Your Own Proxy](https://scrapfly.io/docs/cloud-browser-api/bring-your-own-proxy) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) - [Native Browser MCP](https://scrapfly.io/docs/cloud-browser-api/mcp) - [DevTools Protocol](https://scrapfly.io/docs/cloud-browser-api/cdp-reference) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Vapi [ View as markdown ](https://scrapfly.io/?view=markdown) Copy for LLM Copy for LLM [ Open in ChatGPT ](https://chatgpt.com/?hints=search&prompt=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Fvapi%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Claude ](https://claude.ai/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Fvapi%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Perplexity ](https://www.perplexity.ai/search/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Fvapi%20so%20I%20can%20ask%20questions%20about%20it.) Voice AI platform for building conversational agents. Integrate Scrapfly to enable voice assistants that can fetch live web data during phone calls and voice interactions. AI Voice Platform Cloud REST API Webhooks [ Official Website ](https://vapi.ai/) 1. [MCP Documentation](https://scrapfly.io/docs/mcp) 2. [Integrations](https://scrapfly.io/docs/mcp/integrations) 3. Vapi ## Prerequisites Before getting started, make sure you have the following: - Vapi account ([sign up here](https://vapi.ai/)) - Your Scrapfly API key - Webhook endpoint or server to handle function calls **Note:** Vapi does not directly support MCP servers. Instead, we integrate Scrapfly through [function calling](https://docs.vapi.ai/assistants/function-calling), where your webhook server calls Scrapfly's API when the voice assistant needs web data. ## Setup Instructions Integrate Scrapfly with Vapi by creating function definitions and webhook handlers. This enables your voice assistants to scrape web data in real-time during conversations. 1. **Create Webhook Server** Set up a webhook server to handle Vapi function calls and execute Scrapfly API requests: ### Python Flask Example ``` from flask import Flask, request, jsonify import requests import os app = Flask(__name__) @app.route("/vapi/scrape", methods=["POST"]) def scrape_webpage(): """Handle Vapi function call and execute Scrapfly scraping""" # Get parameters from Vapi data = request.json url = data.get("message", {}).get("functionCall", {}).get("parameters", {}).get("url") if not url: return jsonify({"error": "URL parameter required"}), 400 # Call Scrapfly API scrapfly_response = requests.get("{{ public_api_endpoint }}/scrape", params={ "key": os.getenv("SCRAPFLY_API_KEY"), "url": url, "format": "markdown" }) if scrapfly_response.status_code == 200: content = scrapfly_response.json().get("result", {}).get("content", "") # Return result to Vapi return jsonify({ "result": content[:2000] # Truncate for voice response }) else: return jsonify({ "error": f"Failed to scrape {url}" }), 500 if __name__ == "__main__": app.run(port=5000) ``` ### Node.js Express Example ``` import express from "express"; import axios from "axios"; const app = express(); app.use(express.json()); app.post("/vapi/scrape", async (req, res) => { const url = req.body.message?.functionCall?.parameters?.url; if (!url) { return res.status(400).json({ error: "URL parameter required" }); } try { const response = await axios.get("{{ public_api_endpoint }}/scrape", { params: { key: process.env.SCRAPFLY_API_KEY, url: url, format: "markdown" } }); const content = response.data.result?.content || ""; // Return result to Vapi res.json({ result: content.substring(0, 2000) // Truncate for voice }); } catch (error) { res.status(500).json({ error: `Failed to scrape ${url}` }); } }); app.listen(5000, () => console.log("Server running on port 5000")); ``` Tip: Deploy Your WebhookDeploy your webhook server to a public URL using services like: - **Railway**: Easy deployment with free tier - **Render**: Free web service hosting - **Vercel/Netlify**: Serverless function deployment - **ngrok**: Quick local testing (development only) 2. **Configure Function in Vapi Dashboard** Add a custom function to your Vapi assistant that calls your webhook: 1. Log in to [Vapi Dashboard](https://dashboard.vapi.ai/) 2. Navigate to your assistant or create a new one 3. Go to "Functions" section 4. Click "Add Function" 5. Configure the function with this schema: ``` { "name": "scrape_webpage", "description": "Scrape content from a webpage and return it", "parameters": { "type": "object", "properties": { "url": { "type": "string", "description": "The URL to scrape" } }, "required": ["url"] }, "server": { "url": "https://your-webhook-url.com/vapi/scrape", "method": "POST" } } ``` **Important:** Replace `https://your-webhook-url.com` with your actual webhook server URL. 3. **Configure Assistant Instructions** Update your Vapi assistant's system prompt to use the scraping function: ``` You are a helpful voice assistant that can look up information from the web in real-time. When a user asks for information about a specific webpage or URL, use the scrape_webpage function to fetch the latest content. Example conversations: - User: "What's on the Hacker News homepage?" You: Call scrape_webpage(url="https://news.ycombinator.com") and summarize the results - User: "Check the price on https://web-scraping.dev/product" You: Call scrape_webpage(url="https://web-scraping.dev/product") and extract the price Always confirm the URL with the user before scraping, and summarize the key information concisely for voice delivery. ``` 4. **Test Your Voice Assistant** Test the integration using Vapi's dashboard or API: 1. Start a test call in Vapi Dashboard 2. Ask the assistant to look up information: *"What are the top posts on Hacker News?"* 3. The assistant should call your webhook, which calls Scrapfly, and read the results **Pro Tip:** Keep scraped content summaries short (under 500 characters) for natural voice delivery! ## Example Prompts ###### Product Information Lookup Check the current price and availability on this product page: [URL] ###### News Headlines What are the top stories on Hacker News right now? ###### Business Hours Lookup Look up the business hours from their website ###### Competitive Pricing Compare the pricing on these competitor websites: [URL1, URL2] ## Troubleshooting ##### Webhook Not Receiving Calls **Problem:** Vapi does not call your webhook server **Solution:** - Verify webhook URL is publicly accessible (test with curl or Postman) - Check function is properly configured in Vapi dashboard - Ensure webhook accepts POST requests - Review Vapi logs in dashboard for error messages ##### Scrapfly API Errors **Problem:** Webhook returns errors from Scrapfly API **Solution:** - Verify SCRAPFLY\_API\_KEY environment variable is set correctly - Check URL format is valid (must start with http:// or https://) - Ensure sufficient Scrapfly credits available - Review Scrapfly API response for specific error details ##### Voice Responses Too Long **Problem:** Scraped content is too lengthy for voice delivery **Solution:** - Truncate content to ~500 characters max - Use AI summarization before returning results - Extract only key information (prices, titles, dates) - Format responses as bullet points for clarity ##### Webhook Timeouts **Problem:** Vapi times out waiting for webhook response **Solution:** - Optimize webhook code for faster response times - Cache frequently scraped URLs - Use Scrapfly cache parameter for repeat requests - Return partial results quickly rather than waiting for full scrape ##### Function Not Being Called **Problem:** Assistant does not use scrape\_webpage function **Solution:** - Update assistant instructions to explicitly mention using the function - Use clearer prompts mentioning "look up", "check", or "scrape" - Verify function schema is correct in Vapi dashboard - Test with explicit URLs in user prompts ## Next Steps - [Explore available MCP tools](https://scrapfly.io/docs/mcp/tools) and their capabilities - [See real-world examples](https://scrapfly.io/docs/mcp/examples) of what you can build - [Learn about authentication methods](https://scrapfly.io/docs/mcp/authentication) in detail - [Read the FAQ](https://scrapfly.io/docs/mcp/faq) for common questions [ Back to All Integrations ](https://scrapfly.io/docs/mcp/integrations)