Scrapfly CrewAI Integration

View as markdown

Copy for LLM

Copy for LLM Open in ChatGPT Open in Claude Open in Perplexity

Scrapfly is available on CrewAI - a popular a framework for developing applications powered by large language models (LLMs).

For CrewAI, Scrapfly is available as a CrewAI Tool object which uses Scrapfly Web Scrape API to retrieve web page data for use within the LangChain ecosystem.

Usage

To start get your Srapfly API key on your dashboard. Then install Scrapfly Python SDK, CrewAI and CrewAI Tools:

pip install crewai crewai[tools] scrapfly-sdk[all]

Then, the ScrapflyScrapeWebsiteTool is available for scraping any web page:

import os
from crewai import Agent, Task, Crew
from crewai_tools import ScrapflyScrapeWebsiteTool

scrape_tool = ScrapflyScrapeWebsiteTool("YOUR SCRAPFLY API KEY")

For more advanced use, the integration supports all Scrapfly Web Scrape API options matching the Python SDK signature which can be provided as a natural language parameter:

research = Task(
description="Extract reviews of product page https://web-scraping.dev/product/1 and perform sentiment analysis on each review. \
use scrape_config={\"render_js\":True}",
expected_output="a list of sentiment analysis for each review on the page.",
agent=researcher,
)

Example Use

CrewAI is a framework that works by defining Tasks, Agents, and Crews. A Crew is a group of Agents working together toward a common goal. Each Agent is assigned one or more Tasks, and the Crew orchestrates their collaboration to perform the given tasks. Let's define the required objects. We'll define a scrape task, and agent to perform it, and finally group them in a crew for execution:

import os
from crewai import Agent, Task, Crew
from crewai_tools import ScrapflyScrapeWebsiteTool

# Set the required environment variables
os.environ["OPENAI_API_KEY"] = "Your OpenAI API Key"

# Initialize the tool, Scrapfly
scrape_tool = ScrapflyScrapeWebsiteTool(api_key='Your Scrapfly API Key')


# Define an agent that uses the tool and the LLM
web_scraper_agent = Agent(
    role="Research Analyst",
    goal="Find and summarize information about specific topics",
    backstory="You are an experienced researcher with attention to detail",
    llm="gpt-4", # Specify the LLM model to use
    tools=[scrape_tool], # Add the tool to the agent
    verbose=True  # Enable logging for debugging
)

# Example task to extract content from a website
scrape_task = Task(
    description="Extract the main content from the product page at https://web-scraping.dev/products and summarize the available products . \
    use scrape_config={\"asp\": True, \"render_js\": True, \"proxy_pool\": \"public_residential_pool\", \"country\": \"us\", \"auto_scroll\": True}",
    expected_output="A summary of the products available on the website.",
    agent=web_scraper_agent,
)

# Create the crew with the agent and task
crew = Crew(
    agents=[web_scraper_agent],
    tasks=[scrape_task],
    verbose=True,
)

# Run the task
result = crew.kickoff()

print("Task Result:\n", result)

In the above example, we define a complete CrewAi workflow by combining an agent, a task, and a tool. Let's break down the execution flow:

• The ScrapflyScrapeWebsiteTool is initialized with an API key to handle web scraping.
• An Agent is created with the role of a research analyst, powered with both an LLM (gpt-4) and the scraping tool defined earlier.
• A Task is defined to extract and summarize product information from a target web page with the required scrape configurations from Scrapfly.
• Finally, a Crew is defined by grouping the agent and the task together. When crew.kickoff() is called, the agent executes the task using its tools and LLM, and the final summarized result is returned.

Errors

CrewAI will display the Scrapfly API error message in the standard Scrapfly API error message format. For more see:

• Web Scraping Errors
• Troubleshooting Errors

Pricing

No additional costs.