# SEO & SERP Web Scraping ## Rank tracking, keyword research, and competitor monitoring from live SERPs. Web scraping Search Engine Result Pages is the only reliable source of truth for SEO. Pull live rankings, featured snippets, local packs, and organic positions across Google, Bing, DuckDuckGo, Baidu, and more. [ Get Free API Key ](https://scrapfly.io/register) [ Web Scraping API ](https://scrapfly.io/products/web-scraping-api) 1,000 free credits. No credit card required. --- ## 5+ search engines - Google, Bing, DuckDuckGo, Yandex, Baidu ## 5B+ scrapes / month platform-wide ## 99%+ anti-bot bypass success rate ## JSON or CSV - structured output, ready to ingest --- // FORMULA## Turn every SERP into a tracked ranking. `Query` + `Schema` = Ranked Result Send any search query with a geo target. Get back position, title, URL, snippet, and SERP features as structured data. --- COVERAGE## Everything a SERP Contains From rank tracking to local packs. Every signal, every engine. // FEATURED ### Rank Tracking Poll any keyword at any cadence. Track your domain and competitors across devices, geos, and search engines. Build time-series datasets that reveal how algorithm changes affect visibility. **daily**polling cadence **190+**geo targets **desktop**device type **mobile**device type Google Bing DuckDuckGo ### Keyword Research Discover which keywords land in position 1-3, which trigger knowledge panels, and which bring up shopping carousels. Cross-reference against your content to spot gaps in real time. **organic**results **paid**ads **related**queries ### Featured Snippets & SERP Features Modern SERPs are more than a ranked list. Extract every enriched result type to understand which features appear for your target keywords and who owns them. People Also Ask Featured Snippet Knowledge Panel Shopping Results Image Pack Video Carousel Local Pack News Box ### Competitor Visibility Track which domains appear, how often, and at what position across your keyword set. Calculate share-of-voice per competitor and feed it into your dashboard automatically. **SERP** raw page, all result types extracted **Domains** which competitors appear and at what position **Share of Voice** percentage of visible clicks across keyword set **Dashboard** time-series ready JSON for your BI pipeline ### Local SERPs Search results differ by city, region, and country. Scrapfly's geo-targeted proxy network lets you pull SERPs as a real user in any location, so your local rank data is accurate. United States United Kingdom Germany France Japan 190+ more ### Anti-bot Bypass Included Search engines detect and block scraper traffic aggressively. Scrapfly handles fingerprinting, CAPTCHA, and proxy rotation automatically so your rank tracking pipeline never goes dark. [Cloudflare](https://scrapfly.io/bypass/cloudflare) [DataDome](https://scrapfly.io/bypass/datadome) [Akamai](https://scrapfly.io/bypass/akamai) [PerimeterX](https://scrapfly.io/bypass/perimeterx) [See full bypass coverage](https://scrapfly.io/bypass) --- - Web Scraping API - Extraction API - Screenshot API - Crawler API - Cloud Browser Products ## One Key. Every SEO Data Primitive. Scrape, render, extract, and crawl - all managed behind a single API. Web Scraping API Fetch any SERP URL with anti-bot bypass, geo-targeted proxies, and optional JS rendering. Returns clean HTML or structured JSON ready for parsing. $> `POST https://api.scrapfly.io/scrape` [ Landing page ](https://scrapfly.io/products/web-scraping-api) [ Documentation ](https://scrapfly.io/docs/scrape-api/getting-started) Extraction API Turn raw SERP HTML into structured rank data with a prompt or JSON schema. LLM-powered, built-in templates for search results, organic listings, and ads. $> `POST https://api.scrapfly.io/extraction` [ Landing page ](https://scrapfly.io/products/extraction-api) [ Documentation ](https://scrapfly.io/docs/extraction-api/getting-started) Screenshot API Capture full-page SERP screenshots for visual audits, change detection, and documentation. PNG, JPEG, or WebP with custom viewport. $> `POST https://api.scrapfly.io/screenshot` [ Landing page ](https://scrapfly.io/products/screenshot-api) [ Documentation ](https://scrapfly.io/docs/screenshot-api/getting-started) Crawler API Traverse search result pages, follow pagination, and discover linked pages at depth. Every URL runs through the Web Scraping API automatically. $> `POST https://api.scrapfly.io/crawler` [ Landing page ](https://scrapfly.io/products/crawler-api) [ Documentation ](https://scrapfly.io/docs/crawler-api/getting-started) Cloud Browser Drive a real stealth Chromium over CDP for JS-heavy search engines or dynamic SERP variants. Full Playwright and Puppeteer compatibility. $> `wss://browser.scrapfly.io/cdp?key=...` [ Landing page ](https://scrapfly.io/products/cloud-browser-api) [ Documentation ](https://scrapfly.io/docs/cloud-browser-api/getting-started) [Get Free API Key](https://scrapfly.io/register) --- CODE## SERP Data Made Easy Start scraping search engines in minutes with any language. Anti-bot bypass, JS rendering, and geo-targeting on a real Google search results page. Python TypeScript HTTP / cURL ``` from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse client = ScrapflyClient(key="API KEY") api_response: ScrapeApiResponse = client.scrape( ScrapeConfig( # add a page to scrape url='https://www.google.com/search?q=scrapfly', asp=True, # enable bypass of anti-scraping protection render_js=True, # enable headless browser (if necessary) country="US", # set location for region specific data # use AI to extract data extraction_model='search_engine_results' ) ) # use AI extracted data print(api_response.scrape_result['extracted_data']['data']) # or parse the html yourself print(api_response.content) ``` ``` import { ScrapflyClient, ScrapeConfig } from 'jsr:@scrapfly/scrapfly-sdk'; const client = new ScrapflyClient({ key: "API KEY" }); let api_response = await client.scrape( new ScrapeConfig({ // add a scrape url url: 'https://www.google.com/search?q=scrapfly', asp: true, // enable bypass of anti-scraping protection render_js: true, // enable headless browser (if necessary) // use AI to extract data extraction_model: 'search_engine_results' }) ); // use AI extracted data console.log(api_response.result['extracted_data']['data']) // or parse the HTML yourself console.log(api_response.result['content']) ``` ``` http https://api.scrapfly.io/scrape \ key==$SCRAPFLY_KEY \ url==https://www.google.com/search?q=scrapfly \ asp==true \ render_js==true \ country==US \ extraction_model=search_engine_results ``` [ Python SDK docs → ](https://scrapfly.io/docs/sdk/python) [ TypeScript SDK docs → ](https://scrapfly.io/docs/sdk/typescript) [ HTTP API docs → ](https://scrapfly.io/docs) --- AGENT MODE## Automate with AI & Workflows Connect Scrapfly to your AI agents, LLM pipelines, and no-code tools for fully automated SEO monitoring. ### MCP Server Expose Scrapfly's scrape, extract, and screenshot capabilities as tool calls for Claude, GPT-4, and any MCP-compatible agent. Build fully automated SERP monitoring pipelines without writing glue code. [Explore MCP Cloud](https://scrapfly.io/products/mcp-cloud) ### Workflow Automation Use Scrapfly inside n8n, Zapier, or Make to schedule daily rank checks, trigger alerts when positions change, and push structured SERP data directly into your data warehouse. n8n Zapier Make ### Python & TypeScript SDKs Official SDKs with full SERP scraping support. Async-ready, type-safe, and shipping with retry, concurrency, and error-handling built in so your rank tracker stays up. [Python SDK](https://scrapfly.io/docs/sdk/python) [TypeScript SDK](https://scrapfly.io/docs/sdk/typescript) --- FAQ## Frequently Asked Questions ### HOW TO UNBLOCK ACCESS TO SEARCH ENGINE WEBSITES? While scraping search engines is perfectly legal, search engines detect and block automated traffic. You can harden your scraper yourself using techniques covered in our [anti-bot bypass guide](https://scrapfly.io/blog/posts/how-to-scrape-without-getting-blocked-tutorial/), or delegate the entire problem to the [Web Scraping API](https://scrapfly.io/products/web-scraping-api) which handles fingerprinting, CAPTCHA, and proxy rotation automatically. ### IS WEB SCRAPING SEARCH ENGINE DATA LEGAL? Yes, scraping publicly visible search result data is generally legal in most jurisdictions. It is best practice to avoid collecting Personally Identifiable Information. For a detailed breakdown, see our [web scraping laws](https://scrapfly.io/is-web-scraping-legal) article. ### WHAT SEO DATA CAN BE SCRAPED FROM SERPS? You can extract organic rankings, paid ads, featured snippets, People Also Ask boxes, knowledge panels, local packs, shopping results, image packs, and related search queries. Combining these signals gives a complete picture of SERP ownership for any keyword. ### WHAT IS A WEB SCRAPING API? A [Web Scraping API](https://scrapfly.io/products/web-scraping-api) abstracts away the complexities of fetching web pages reliably at scale. It handles anti-bot bypass, proxy rotation, JS rendering, and retry logic so your application receives clean page content without building that infrastructure yourself. ### HOW CAN I ACCESS THE WEB SCRAPING API? The API accepts standard HTTP requests from any client - cURL, httpie, or any HTTP library. First-class support is available via the official [Python SDK](https://scrapfly.io/docs/sdk/python) and [TypeScript SDK](https://scrapfly.io/docs/sdk/typescript). ### ARE PROXIES ENOUGH TO SCRAPE SEARCH ENGINE DATA? No. Search engines identify proxy traffic through TLS fingerprints, behavioral signals, and rate patterns. Passing requires combining geo-targeted proxies with browser fingerprint alignment and CAPTCHA handling. Scrapfly bundles all three so you do not have to build or maintain that stack yourself. ### HOW TO EXTRACT STRUCTURED DATA FROM SERPS? SERP HTML changes frequently and is difficult to parse with fixed CSS selectors. The [Extraction API](https://scrapfly.io/products/extraction-api) uses LLM-powered models to pull exactly the fields you need - positions, titles, URLs, snippets, and SERP features - from raw HTML without a brittle parser. --- // GET STARTED### Start tracking rankings in under a minute. Free account, 1,000 credits, no credit card. Anti-bot bypass, geo-targeting, and AI extraction all included. [ Get Free API Key ](https://scrapfly.io/register) [See all use cases](https://scrapfly.io/use-case/web-scraping)