Scrapfly Developer Hub

Build powerful web scraping solutions with our comprehensive APIs. From simple scraping to complex data extraction at scale.

NEW

MCP Server

Give your AI the power to act - Connect AI models to live web data through Model Context Protocol with OAuth2 security and zero configuration.

Key Features:

Zero configuration
OAuth2 security
Natural language queries
IDE integration

View Documentation

# Add to claude_desktop_config.json or cursor settings
{
  "mcpServers": {
    "scrapfly": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://mcp.scrapfly.io/mcp?key="
      ]
    }
  }
}

# Then just ask your AI in natural language:
# "Get me the top headlines from Hacker News"
# "Compare iPhone prices across Amazon, Best Buy, and Walmart"

Web Scraping API

Scrape any page, bypass blocking, control a real web browser and get output results in HTML, Markdown, or JSON formats.

Key Features:

Anti-bot bypass
JavaScript rendering
Multiple output formats
Proxy rotation

View Documentation

# pip install scrapfly-sdk
from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse

scrapfly = ScrapflyClient(key='')

api_response: ScrapeApiResponse = scrapfly.scrape(
    ScrapeConfig(
        url='https://web-scraping.dev/product/1',
        # enable cloud browsers
        render_js=True,
        # bypass anti-scraping protection
        asp=True
        # much more!
    )
)

print(api_response.selector.css('p.product-description::text').get())

Screenshot API

Screenshot capture made easy - capture any area, full page, with auto-scrolling and more advanced options.

Key Features:

Full page screenshots
Element targeting
Auto-scroll support
Multiple formats

View Documentation

# pip install scrapfly-sdk
from scrapfly import ScrapflyClient, ScreenshotConfig

scrapfly = ScrapflyClient(key='')

api_response = scrapfly.screenshot(
    ScreenshotConfig(
        url='https://web-scraping.dev/products',
        # automatically scroll
        auto_scroll=True,
        # capture specific areas
        capture=".products-wrap",
))

with open("screenshot.png", "wb") as f:
    f.write(api_response.image)

Extraction API

Extract data from documents using AI, LLMs and extraction templates - turn unstructured data into structured insights.

Key Features:

AI-powered extraction
LLM integration
Custom templates
Structured output

View Documentation

# pip install scrapfly-sdk
from scrapfly import ScrapeConfig, ExtractionConfig, ScrapflyClient, ScrapeApiResponse, ExtractionApiResponse

scrapfly = ScrapflyClient(key='')

# First, scrape the web page to retrieve its HTML
api_response:ScrapeApiResponse = scrapfly.scrape(scrape_config=ScrapeConfig(
    url='https://web-scraping.dev/products',
    render_js=True
))

html = api_response.content

# Second, pass the HTML and an extraction prompt
# In this example, we'll ask a question about the data
extraction_api_response:ExtractionApiResponse = scrapfly.extract(
    extraction_config=ExtractionConfig(
        body=html, # pass the HTML content
        content_type='text/html', # content data type
        charset='utf-8', # passed content charset, use `auto` if you aren't sure
        extraction_prompt='what is the flavor of the dark energy potion?' # LLM extraction prompt
    )
)

print(extraction_api_response.extraction_result)

Crawler API

Crawl entire websites recursively - extract data from thousands of pages with a single API call efficiently.

Key Features:

Recursive crawling
Bulk data extraction
Content formats
Job management

View Documentation

# pip install scrapfly-sdk
from scrapfly import ScrapflyClient, CrawlConfig

scrapfly = ScrapflyClient(key='')

# Start a crawler job
crawler = scrapfly.crawl(
    CrawlConfig(
        url='https://web-scraping.dev/products',
        page_limit=10,
        # extract content in multiple formats
        content_formats=['markdown', 'html']
    )
)

# Poll for completion
crawler.wait()

# Retrieve results
for url, content in crawler.get_contents():
    print(f"Crawled: {url}")
    print(content['markdown'])

Proxy Saver

Power up your proxies with bandwidth, speed and stability improvements for optimal performance.

Key Features:

Bandwidth optimization
Speed enhancement
Connection stability
Fingerprint management

View Documentation

curl \
--insecure \
--proxy http://proxyId-XXX-FpImpersonate-chrome116:API_KEY@proxy-saver.scrapfly.io:3333 \
https://httpbin.dev/anything

Ready to Get Started?

Choose your preferred language and follow our step-by-step onboarding guide

Start with Python

Start with TypeScript

Integrate with Your Favorite Tools

No-Code Automation

Integrate into your automated workflows with popular platforms:

Zapier Make.com n8n

Developer Libraries

Already have a tech stack? Integrate with popular libraries:

LangChain LlamaIndex CrewAI Scrapy

Official SDKs

Choose your preferred programming language

Python SDK

TypeScript SDK

Type-safe scraping

Scrapy Integration

Powerful framework

Explore with Interactive Examples

Discover intuitive and interactive examples in our playground. Test our solution without writing a single line of code and generate examples in many languages.

What you can do:

Test APIs interactively
Generate code in multiple languages
Debug and troubleshoot
Experiment with parameters

Launch Interactive Player

*Requires an account

Speed Up with Web Scraping Tools

Our internal scraping tools are open for everyone! Use them to test your scrapers, understand bot detection, or just learn everything about web scraping.

Available tools:

HTTP headers analyzer
Browser fingerprint tester
XPath/CSS selector builder
And many more utilities

Discover Web Scraping Tools

Scrapfly Developer Hub

MCP Server

Key Features:

Web Scraping API

Key Features:

Screenshot API

Key Features:

Extraction API

Key Features:

Crawler API

Key Features:

Proxy Saver

Key Features:

Ready to Get Started?

Integrate with Your Favorite Tools

No-Code Automation

Developer Libraries

Official SDKs

Python SDK

TypeScript SDK

Scrapy Integration

Explore with Interactive Examples

What you can do:

Web Scraping Academy

Fundamentals

Anti-Bot Bypass

Real Projects

Optimization

Speed Up with Web Scraping Tools

Available tools: