Scrapfly CrewAI Integration

Scrapfly is available on CrewAI - a popular a framework for developing applications powered by large language models (LLMs).

For CrewAI, Scrapfly is available as a CrewAI Tool object which uses Scrapfly Web Scrape API to retrieve web page data for use within the LangChain ecosystem.

Usage

To start get your Srapfly API key on your dashboard. Then install Scrapfly Python SDK, CrewAI and CrewAI Tools:

pip install crewai crewai[tools] scrapfly-sdk[all]

Then, the ScrapflyScrapeWebsiteTool is available for scraping any web page:

import os
from crewai import Agent, Task, Crew
from crewai_tools import ScrapflyScrapeWebsiteTool

scrape_tool = ScrapflyScrapeWebsiteTool("YOUR SCRAPFLY API KEY")

For more advanced use, the integration supports all Scrapfly Web Scrape API options matching the Python SDK signature which can be provided as a natural language parameter:

research = Task(
description="Extract reviews of product page https://web-scraping.dev/product/1 and perform sentiment analysis on each review. \
use scrape_config={\"render_js\":True}",
expected_output="a list of sentiment analysis for each review on the page.",
agent=researcher,
)

Errors

CrewAI will display the Scrapfly API error message in the standard Scrapfly API error message format. For more see:

• Web Scraping Errors
• Troubleshooting Errors

Pricing

No additional costs.