CrewAI Integration

Power up LLM with web scraping

Scrapfly officially integrates with CrewAI framework for LLM tool development in Python. Making RAG accessible to anyone:

Scrape any page using Web Scraping API and all of its features like cloud web browsers and blocking bypass
RAG extend your CrewAI tools with web scraped documents using Scrapfly document loader
Auto convert scraped data to Markdown, JSON or other data types for easy ingestion

Code Example Documentation

CrewAI Integration web web scraping and automation

Get Started with CrewAI Web Automation

start scraping and data extracting with crewai & scrapfly in 5 minutes

What can CrewAI integration do?

scrape any page and create your index

from crewai import Agent, Task, Crew
from crewai_tools import ScrapflyScrapeWebsiteTool

# Initialize the tool
scrape_tool = ScrapflyScrapeWebsiteTool(api_key="your_scrapfly_api_key")

# Define an agent that uses the tool
web_scraper_agent = Agent(
    role="Web Scraper",
    goal="Extract information from websites",
    backstory="An expert in web scraping who can extract content from any website.",
    tools=[scrape_tool],
    verbose=True,
)

# Example task to extract content from a website
scrape_task = Task(
    description="Extract the main content from the product page at https://web-scraping.dev/products and summarize the available products.",
    expected_output="A summary of the products available on the website.",
    agent=web_scraper_agent,
)

# Create and run the crew
crew = Crew(agents=[web_scraper_agent], tasks=[scrape_task])
result = crew.kickoff()

ScrapflyReader extends CrewAI with the ability to scrape any page and extend your LLM operations with RAG functionality:

Bypass scraper blocking to collected web page datasets
Use javascript rendering to scrape all data on available the page
Automatically convert results to markdown or json for better LLM understanding

Scrapfly integration handles all of the document retrieval challenges in your LLM applications so you can focus on delivering real AI products.

More Examples

What's included in the integration?

Need more functionality?

Scrapfly is accessible through Python and Typescript SDKs so you can create your own scripts and integrations in Python or any Javascript runtime like NodeJS, Deno or Bun!

The SDKs include all Scrapfly API features and many useful utilities and shortcuts making for a powerful development experience.

Python SDK

Typescript SDK

Python, NodeJS, Deno, Bun

Transform Your Industry with Web Data

Explore web data solutions for your industry — we got you covered!

AI Training

Crawl the latest images, videos and user generated content for AI training.

Instagram Youtube

Compliance

Scrape online presence to validate compliance and security.

Github Pastebin

eCommerce

Scrape products, reviews and more to enhance your eCommerce and brand awareness.

Amazon eBay

Financial Service

Scrape the latest stock, shipping and financial data to enhance your finance datasets.

Marketwatch wsj

Fraud Detection

Scrape products and listings to detect fraud and counterfeit activity.

eBay Craigslist

Jobs Data

Scrape the latest job listings, salaries and more to enhance your job search.

Glassdoor Indeed

Lead Generation

Scrape online profiles and contact details to enhance your lead generation.

LinkedIn Crunchbase

Logistics

Scrape logistics data like shipping, tracking, container prices to enhance your deliveries.

Maersk Hapag-lloyd

Explore More Use Cases

Frequently Asked Questions

How to Web Scrape with CrewAI?

Creating a CrewAI web scraper is available using its external tools, which are injectable utilities. ScrapflyScrapeWebsiteTool represents Scrapfly's implementation of such a scrape utility. It enables scraping any web page while bypassing its anti-bot protection to return its data as HTML, Text, or Markdown. The data retrieved are then used to power a CrewAI LLM agent.

How CrewAI Works?

CrewAI is a framework for building AI agents that work like a "crew", each with defined roles and goals. Crews optimize for autonomy and collaborative intelligence, allowing agents to coordinate naturally and complete complex tasks. Flows provide event-driven, low-level control with precise orchestration through single LLM calls, while still supporting Crews natively.

Is Web Scraping with CrewAI Legal?

Yes, generally using CrewAI for web scraping public data is legal around the globe. However, attention should be paid to scraping PII (personally identifiable information) and any copyrighted material that may have storage restrictions due to regulations like the GDPR. For more, see our in-depth web scraping laws article.

What is a Web Scraping API?

Web Scraping API is a service that abstracts away the complexities and challenges of web scraping and data extraction. This allows developers to focus on creating software rather than dealing with issues like web scraping blocking and other data access challenges.

What is an Extraction API?

Extraction API is a service that abstracts away the complexities and challenges of data extraction and parsing. It does this through AI auto extract and LLM prompt features as well as manual schema based instructions for precise control

What is an Screenshot API?

Screenshot API is a service that abstracts away the complexities and challenges web browser screenshot capture. This allows you to capture a screenshot of any web page while handling challenges like blocking ads and pop-ups, bypassing browser blocks, and returning the screenshot in any format of any page area you need.