Prerequisites

Before getting started, make sure you have the following:

• Python 3.10+ installed
• crewai and crewai-tools packages installed
• Your Scrapfly API key (only if not using OAuth2)

Setup Instructions

CrewAI supports MCP tools through the tools integration. Follow these steps to equip your agent crew with Scrapfly capabilities.

1. Install Required Packages

   Install CrewAI and the tools package with MCP support:

   pip install crewai crewai-tools
   Tip: Virtual Environment

   Use a virtual environment to manage dependencies:

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
   pip install crewai crewai-tools
2. Initialize Scrapfly MCP Tools

   Select your preferred authentication method:

   from crewai_tools import MCPTool
   
   # Initialize Scrapfly MCP tools with OAuth2
   scrapfly_tools = MCPTool.from_server(
       name="scrapfly",
       command="npx",
       args=["mcp-remote", "https://mcp.scrapfly.io/mcp"]
   )
   
   print(f"Loaded {len(scrapfly_tools)} Scrapfly tools")
   for tool in scrapfly_tools:
       print(f"  - {tool.name}")

   How it works: Your application displays an OAuth2 URL in the console. Open it in your browser to authorize - happens once, then managed automatically. See project-scoped setup for team collaboration.

   from crewai_tools import MCPTool
   
   # Initialize Scrapfly MCP tools with API key
   scrapfly_tools = MCPTool.from_server(
       name="scrapfly",
       command="npx",
       args=["mcp-remote", "https://mcp.scrapfly.io/mcp?key="]
   )
   
   print(f"Loaded {len(scrapfly_tools)} Scrapfly tools")

   Sign up for free to get your API key.
3. Create Agents with Scrapfly Tools

   Build a CrewAI agent equipped with web scraping capabilities:

   from crewai import Agent, Task, Crew
   from crewai_tools import MCPTool
   
   # Load Scrapfly MCP tools
   scrapfly_tools = MCPTool.from_server(
       name="scrapfly",
       command="npx",
       args=["mcp-remote", "https://mcp.scrapfly.io/mcp"]
   )
   
   # Create a research agent with Scrapfly tools
   researcher = Agent(
       role="Web Research Specialist",
       goal="Gather accurate data from the web using Scrapfly",
       backstory="Expert at finding and extracting web data efficiently",
       tools=scrapfly_tools,
       verbose=True
   )
   
   # Create a task for the agent
   research_task = Task(
       description="Scrape the top 10 posts from Hacker News and create a summary",
       expected_output="A markdown summary of the top posts with titles and URLs",
       agent=researcher
   )
   
   # Create and run the crew
   crew = Crew(
       agents=[researcher],
       tasks=[research_task],
       verbose=True
   )
   
   result = crew.kickoff()
   print(result)
   Pro Tip: The agent will automatically call scraping_instruction_enhanced to get required parameters before scraping!
4. Build a Multi-Agent Crew

   Create a team of agents that collaborate on web scraping and data analysis:

   from crewai import Agent, Task, Crew
   from crewai_tools import MCPTool
   
   # Load Scrapfly tools
   scrapfly_tools = MCPTool.from_server(
       name="scrapfly",
       command="npx",
       args=["mcp-remote", "https://mcp.scrapfly.io/mcp"]
   )
   
   # Data Collector Agent
   collector = Agent(
       role="Data Collector",
       goal="Scrape web data efficiently",
       backstory="Specialist in web scraping and data extraction",
       tools=scrapfly_tools,
       verbose=True
   )
   
   # Data Analyst Agent
   analyst = Agent(
       role="Data Analyst",
       goal="Analyze and summarize collected data",
       backstory="Expert at finding insights in web data",
       verbose=True
   )
   
   # Tasks
   collection_task = Task(
       description="Scrape product listings from https://web-scraping.dev/products",
       expected_output="Raw product data in JSON format",
       agent=collector
   )
   
   analysis_task = Task(
       description="Analyze the product data and create a pricing comparison report",
       expected_output="A markdown report with pricing insights",
       agent=analyst,
       context=[collection_task]  # Depends on collection_task output
   )
   
   # Create crew with multiple agents
   crew = Crew(
       agents=[collector, analyst],
       tasks=[collection_task, analysis_task],
       verbose=True
   )
   
   result = crew.kickoff()
   print(result)

Example Prompts

Troubleshooting

Next Steps

• Explore available MCP tools and their capabilities
• See real-world examples of what you can build
• Learn about authentication methods in detail
• Read the FAQ for common questions