# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Bring Your Own Proxy](https://scrapfly.io/docs/cloud-browser-api/bring-your-own-proxy) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # ChatGPT [ View as markdown ](https://scrapfly.io/?view=markdown) Copy for LLM Copy for LLM [ Open in ChatGPT ](https://chatgpt.com/?hints=search&prompt=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Fchatgpt%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Claude ](https://claude.ai/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Fchatgpt%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Perplexity ](https://www.perplexity.ai/search/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fmcp%2Fintegrations%2Fchatgpt%20so%20I%20can%20ask%20questions%20about%20it.) Connect Scrapfly's MCP server to ChatGPT via Developer Mode for AI-powered web scraping, or use our CustomGPT for instant conversational data extraction across the web. AI Assistant Web iOS Android macOS Windows [ Official Website ](https://chat.openai.com/) 1. [MCP Documentation](https://scrapfly.io/docs/mcp) 2. [Integrations](https://scrapfly.io/docs/mcp/integrations) 3. ChatGPT ## Prerequisites Before getting started, make sure you have the following: - [ChatGPT Plus or Team subscription](https://chat.openai.com/) - An active [Scrapfly account](https://scrapfly.io/register) - Your Scrapfly API key (from your [Dashboard](https://scrapfly.io/dashboard)) ## Getting Started Watch this quick video tutorial to get up and running with Scrapfly MCP on ChatGPT: ## Setup via MCP (Developer Mode) Connect Scrapfly MCP directly to ChatGPT using Developer Mode. This gives you full access to all Scrapfly scraping tools with your own API key. 1. **Navigate to App Settings**In ChatGPT, click your **profile icon** in the bottom-left corner and select **Settings**. In the left-hand menu, click on **Apps**, then select **Advanced settings**. 2. **Enable Developer Mode**Toggle **Developer mode** to the ON position, then click the **"Create app"** button at the top right of the window. 3. **Connect Scrapfly MCP**Fill in the following details in the configuration window: - **Name:** `Scrapfly` - **MCP Server URL:** ``` https://mcp.scrapfly.io/mcp?key= ``` - **Authentication:** Select `No Auth` (authentication is handled securely via the API key in the URL) - **Mandatory:** Check the box *"I understand and want to continue"* to acknowledge the use of a custom MCP server Click **"Create"**. [Sign up for free](https://scrapfly.io/register) to get your API key. 4. **Start scraping!**You are all set! Simply ask ChatGPT to scrape a webpage in your normal chat window. ChatGPT will automatically trigger the Scrapfly tool when needed to retrieve real-time data. **Note:** The first time you prompt a scrape, ChatGPT will ask you to click **"Allow"** to grant Scrapfly access. ## Alternative: CustomGPT Scrapfly also provides a CustomGPT that integrates web scraping capabilities directly into ChatGPT. This enables conversational, AI-powered data collection where you simply describe what data you need, and ChatGPT handles the technical scraping details. **Try the Scrapfly CustomGPT:** Available to ChatGPT Plus and Team subscribers. No additional setup required - start scraping conversationally! ## What You Can Do The Scrapfly CustomGPT enables a wide range of conversational web scraping use cases: ##### Research & Analysis - Market research and competitor analysis - Industry trend monitoring - Academic and scientific research - News and media monitoring ##### Content Extraction - Article and blog post scraping - Product information extraction - Review and rating collection - Technical documentation mining ##### Data Monitoring - Price tracking and comparison - Availability checking - Change detection - Alert generation ##### Visual Capture - Full-page screenshots - Multi-viewport captures - Element-specific screenshots - Responsive design testing ## How It Works The CustomGPT uses OpenAI's function calling to access Scrapfly's web scraping infrastructure. When you ask for data from a website: 1. **You describe what you need**Use natural language to explain what data you want to extract from which website. 2. **ChatGPT calls Scrapfly's API**The CustomGPT automatically invokes the appropriate Scrapfly scraping functions with optimal parameters. 3. **Data is extracted and formatted**Scrapfly scrapes the website, bypassing anti-bot protections, and returns clean data. 4. **ChatGPT presents results**The AI analyzes, formats, and presents the data according to your request - tables, summaries, insights, etc. **No coding required:** The CustomGPT handles all technical details - selectors, parsing, error handling, and anti-bot bypass. You just describe what you need! ## Example Conversations Here are real-world examples of conversational scraping with the Scrapfly CustomGPT: ### Research & Analysis ###### Market Research Compare the pricing of the top 3 web scraping services and create a detailed comparison table ###### Competitor Intelligence Analyze competitor.com blog, extract all article titles from the last 6 months, and identify trending topics ###### Industry News Scrape Hacker News front page, filter for AI-related stories, and summarize the top discussions ### Content Extraction ###### Article Scraping Extract the full text, author, and publish date from this Medium article and format as markdown ###### Product Data Scrape this Amazon product page and extract the price, rating, review count, and top 5 customer reviews ###### Documentation Mining Extract all API endpoint names and descriptions from stripe.com/docs/api and create a reference table ### Data Monitoring ###### Price Comparison Check the current price of iPhone 15 on Apple.com, Amazon, and Best Buy, then create a comparison ###### Availability Tracking Check if the Nintendo Switch is in stock at these 5 retailer URLs and report availability status ###### Change Detection Compare the current pricing page of competitor.com with the version from last month and highlight changes ### Screenshots & Visual Analysis ###### Full Page Screenshot Take a full-page screenshot of scrapfly.io homepage and analyze the design layout ###### Responsive Design Capture screenshots of this landing page in mobile, tablet, and desktop viewports and compare layouts ###### Visual Comparison Screenshot the pricing sections of Scrapfly, ScrapingBee, and Zyte, then compare their design approaches ### Multi-Step Workflows ###### News Aggregation Find the top 5 AI-related posts on Hacker News, scrape each article, and create a summary digest with key points ###### Product Research Get today's top 3 products from Product Hunt, scrape their websites, and analyze their value propositions and pricing strategies ###### SEO Analysis Scrape the top 10 Google results for "web scraping API", extract titles and meta descriptions, and analyze SEO patterns ## Limitations & Best Practices ### Current Limitations - **Authentication:** The CustomGPT works without Scrapfly credentials for most public websites, but authenticated scraping requires your own API key - **Rate Limits:** Free usage has rate limits; for high-volume scraping, use your own Scrapfly API key - **Complex Scraping:** For advanced scenarios (JavaScript rendering, CAPTCHA solving, session management), direct API access via [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) or [OpenAI API](https://scrapfly.io/docs/mcp/integrations/openai) is recommended ### Best Practices - **Be Specific:** Clearly describe what data you need and in what format - **Provide URLs:** Include full URLs for the pages you want to scrape - **Iterate:** If results aren't perfect, refine your request with additional details - **Respect Limits:** For high-volume scraping, use the official API with your own credentials ## More Powerful Alternatives For advanced scraping needs, consider these alternatives with full MCP support: ##### [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) Full MCP integration with OAuth2 authentication and unlimited scraping capabilities. - Native MCP protocol support - OAuth2 + API key authentication - Unlimited scraping with your credits - Advanced JavaScript rendering support ##### [OpenAI API](https://scrapfly.io/docs/mcp/integrations/openai) Programmatic access with function calling for custom AI applications. - Full API control and customization - Function calling integration - Build custom assistants and chatbots - Production-ready scalability ## Troubleshooting ##### CustomGPT not accessible **Cause:** CustomGPTs require ChatGPT Plus or Team subscription **Solution:** - Verify you have a ChatGPT Plus or Team subscription - Check the CustomGPT link is not broken (click the link above) - Try accessing from https://chat.openai.com/gpts/discovery - If unavailable, use [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) or [OpenAI API](https://scrapfly.io/docs/mcp/integrations/openai) as alternatives ##### Scraping fails or returns errors **Cause:** Rate limits, invalid URL, or website blocking **Solution:** - Verify the URL is correct and publicly accessible - Check if you've hit rate limits (wait and retry) - For high-volume scraping, use your own Scrapfly API key - Some websites may require JavaScript rendering - specify this in your request - For protected sites, consider [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) with full MCP support ##### Data extraction incomplete or incorrect **Cause:** Ambiguous request or complex page structure **Solution:** - Be more specific about what data you want to extract - Provide examples of the expected output format - Try breaking complex requests into smaller steps - If extraction fails repeatedly, check the page source manually - For dynamic content, specify that JavaScript rendering is needed ##### Response too slow or timeouts **Cause:** Large pages, slow target websites, or heavy processing **Solution:** - Request smaller chunks of data instead of entire pages - Avoid scraping multiple pages in a single request - For batch operations, break into separate conversations - For time-sensitive scraping, use direct API access instead ## Next Steps - [Explore available MCP tools](https://scrapfly.io/docs/mcp/tools) and their capabilities - [See real-world examples](https://scrapfly.io/docs/mcp/examples) of what you can build - [Learn about authentication methods](https://scrapfly.io/docs/mcp/authentication) in detail - [Read the FAQ](https://scrapfly.io/docs/mcp/faq) for common questions [ Back to All Integrations ](https://scrapfly.io/docs/mcp/integrations)