Prerequisites

Before getting started, make sure you have the following:

• Vapi account (sign up here)
• Your Scrapfly API key
• Webhook endpoint or server to handle function calls

Setup Instructions

Integrate Scrapfly with Vapi by creating function definitions and webhook handlers. This enables your voice assistants to scrape web data in real-time during conversations.

1. Create Webhook Server

   Set up a webhook server to handle Vapi function calls and execute Scrapfly API requests:

   Python Flask Example

   from flask import Flask, request, jsonify
   import requests
   import os
   
   app = Flask(__name__)
   
   @app.route("/vapi/scrape", methods=["POST"])
   def scrape_webpage():
       """Handle Vapi function call and execute Scrapfly scraping"""
   
       # Get parameters from Vapi
       data = request.json
       url = data.get("message", {}).get("functionCall", {}).get("parameters", {}).get("url")
   
       if not url:
           return jsonify({"error": "URL parameter required"}), 400
   
       # Call Scrapfly API
       scrapfly_response = requests.get("{{ public_api_endpoint }}/scrape", params={
           "key": os.getenv("SCRAPFLY_API_KEY"),
           "url": url,
           "format": "markdown"
       })
   
       if scrapfly_response.status_code == 200:
           content = scrapfly_response.json().get("result", {}).get("content", "")
           # Return result to Vapi
           return jsonify({
               "result": content[:2000]  # Truncate for voice response
           })
       else:
           return jsonify({
               "error": f"Failed to scrape {url}"
           }), 500
   
   if __name__ == "__main__":
       app.run(port=5000)

   Node.js Express Example

   import express from "express";
   import axios from "axios";
   
   const app = express();
   app.use(express.json());
   
   app.post("/vapi/scrape", async (req, res) => {
     const url = req.body.message?.functionCall?.parameters?.url;
   
     if (!url) {
       return res.status(400).json({ error: "URL parameter required" });
     }
   
     try {
       const response = await axios.get("{{ public_api_endpoint }}/scrape", {
         params: {
           key: process.env.SCRAPFLY_API_KEY,
           url: url,
           format: "markdown"
         }
       });
   
       const content = response.data.result?.content || "";
   
       // Return result to Vapi
       res.json({
         result: content.substring(0, 2000) // Truncate for voice
       });
     } catch (error) {
       res.status(500).json({
         error: `Failed to scrape ${url}`
       });
     }
   });
   
   app.listen(5000, () => console.log("Server running on port 5000"));
   Tip: Deploy Your Webhook

   Deploy your webhook server to a public URL using services like:

   • Railway: Easy deployment with free tier
   • Render: Free web service hosting
   • Vercel/Netlify: Serverless function deployment
   • ngrok: Quick local testing (development only)
2. Configure Function in Vapi Dashboard

   Add a custom function to your Vapi assistant that calls your webhook:

   1. Log in to Vapi Dashboard
   2. Navigate to your assistant or create a new one
   3. Go to "Functions" section
   4. Click "Add Function"
   5. Configure the function with this schema:

   {
     "name": "scrape_webpage",
     "description": "Scrape content from a webpage and return it",
     "parameters": {
       "type": "object",
       "properties": {
         "url": {
           "type": "string",
           "description": "The URL to scrape"
         }
       },
       "required": ["url"]
     },
     "server": {
       "url": "https://your-webhook-url.com/vapi/scrape",
       "method": "POST"
     }
   }
   Important: Replace https://your-webhook-url.com with your actual webhook server URL.
3. Configure Assistant Instructions

   Update your Vapi assistant's system prompt to use the scraping function:

   You are a helpful voice assistant that can look up information from the web in real-time.
   
   When a user asks for information about a specific webpage or URL, use the scrape_webpage function to fetch the latest content.
   
   Example conversations:
   - User: "What's on the Hacker News homepage?"
     You: Call scrape_webpage(url="https://news.ycombinator.com") and summarize the results
   
   - User: "Check the price on https://web-scraping.dev/product"
     You: Call scrape_webpage(url="https://web-scraping.dev/product") and extract the price
   
   Always confirm the URL with the user before scraping, and summarize the key information concisely for voice delivery.
4. Test Your Voice Assistant

   Test the integration using Vapi's dashboard or API:

   1. Start a test call in Vapi Dashboard
   2. Ask the assistant to look up information: "What are the top posts on Hacker News?"
   3. The assistant should call your webhook, which calls Scrapfly, and read the results
   Pro Tip: Keep scraped content summaries short (under 500 characters) for natural voice delivery!

Example Prompts

Troubleshooting

Next Steps

• Explore available MCP tools and their capabilities
• See real-world examples of what you can build
• Learn about authentication methods in detail
• Read the FAQ for common questions