{"items":[{"path":"/blog/posts/how-to-scrape-loopnet","type":"article","title":"How to Scrape LoopNet Commercial Real Estate Listings","description":"Scrape LoopNet commercial real estate listings with Python. Fetch past Akamai, parse JSON-LD, and normalize sale, lease, and auction data.","tags":["blocking","data-parsing","python","scrapeguide"]},{"path":"/blog/posts/best-python-html-parsers","type":"article","title":"7 Best Python HTML Parsers for Web Scraping in 2026","description":"Compare seven Python HTML parsers by CSS and XPath support, malformed HTML behavior, API style, dependencies, and measured parse time.","tags":["css-selectors","data-parsing","python"]},{"path":"/blog/posts/nodriver-tutorial","type":"article","title":"Nodriver Tutorial: Undetected Chrome Automation in Python","description":"Install Nodriver in Python, scrape with its async API, add authenticated proxies, run it headless or on a server, and fix the errors that stop it in 2026.","tags":["blocking","headless-browser","python"]},{"path":"/blog/posts/best-travel-flight-data-apis","type":"article","title":"6 Best Travel and Flight Data APIs for Developers in 2026","description":"Compare six flight and travel data APIs for status, aircraft tracking, schedules, and live fares in 2026.","tags":["api","python"]},{"path":"/blog/posts/how-to-use-proxies-with-selenium","type":"article","title":"How to Use Proxies with Selenium: Authentication, Rotation, and Residential Proxies","description":"Set up authenticated, rotating and residential proxies in Python Selenium, see which Chrome auth methods still work in 2026, and verify the exit IP.","tags":["proxies","python","selenium"]},{"path":"/blog/posts/how-to-scrape-shopify-stores","type":"article","title":"How to Scrape Shopify Stores for Products, Prices, and Variants","description":"Step-by-step guide to scraping Shopify stores with Python, from products.json detection to variant normalization and JSON-LD fallback.","tags":["python","scrapeguide"]},{"path":"/blog/posts/csharp-httpclient-proxy","type":"article","title":"How to Use a Proxy With C# HttpClient in 2026","description":"Configure authenticated and rotating proxies with C# HttpClient and SocketsHttpHandler, then diagnose 407 errors and connection-pool mistakes.","tags":["frameworks","http","proxies"]},{"path":"/blog/posts/how-to-test-web-scraper-before-production","type":"article","title":"How to Test a Web Scraper Before Production","description":"Build parser tests, HTML fixtures, live checks, and deployment gates that catch silent web scraper failures before they reach production.","tags":["data-parsing","python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-idealo","type":"article","title":"How to Scrape idealo: Product Offers, Prices, and Price History","description":"Scrape idealo offers, prices, and price history with Python. Extract the full merchant offer list per product and bypass Akamai with Scrapfly.","tags":["data-parsing","hidden-api","python","scrapeguide"]},{"path":"/blog/posts/best-rust-web-scraping-libraries","type":"article","title":"5 Top Rust Web Scraping Libraries in 2026","description":"Compare five Rust web scraping libraries for HTTP retrieval, HTML parsing, article extraction, and browser automation in 2026, plus what to do about blocked pages.","tags":["rust","tools","web-scraping"]},{"path":"/blog/posts/best-go-web-scraping-libraries","type":"article","title":"9 Best Go (Golang) Web Scraping Libraries and APIs in 2026","description":"Compare nine Go web scraping libraries and APIs for blocked pages, HTTP fetching, HTML parsing, crawling, and browser automation in 2026.","tags":["data-parsing","golang","tools"]},{"path":"/blog/posts/best-puppeteer-alternatives-for-web-scraping","type":"article","title":"4 Best Puppeteer Alternatives for Web Scraping in 2026","description":"Compare 4 open-source Puppeteer alternatives for web scraping in 2026, Playwright, Crawlee, Cheerio and Selenium, plus a cloud browser that runs your code on protected sites.","tags":["cloud-browser","headless-browser","puppeteer"]},{"path":"/blog/posts/how-to-scrape-pinterest","type":"article","title":"How to Scrape Pinterest in 2026","description":"Scrape Pinterest pins, boards, profiles, and images in 2026. Call the hidden resource API, follow its bookmark pagination, and scale past anti-bot blocks.","tags":["hidden-api","python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-goodreads","type":"article","title":"How to Scrape Goodreads for Book Data, Ratings, and Reviews","description":"Learn how to scrape Goodreads for book data, ratings, and reviews with Python, using JSON-LD, embedded Next.js state, and list and search pages.","tags":["hidden-api","python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-news-articles","type":"article","title":"How to Scrape News Articles from Any Website","description":"A 2026 guide to scraping news articles with Python. Extract headline, author, date, and body with Newspaper4k, and get past JS rendering and anti-bot blocks.","tags":["python","scrapeguide"]},{"path":"/blog/answers/persist-sessions-across-scrapy-requests","type":"answer","title":"How to Persist Sessions Across Scrapy Requests","description":"Learn how Scrapy keeps cookies within a crawl, isolates cookie jars, resumes JOBDIR state, and handles Playwright authentication across restarts.","tags":["http","scrapy"]},{"path":"/blog/posts/best-twitter-scrapers-github","type":"article","title":"Top 4 (Twitter) X Scrapers on GitHub That Still Work in 2026","description":"Compare four open-source X scrapers that still work in 2026, including account-based libraries and Scrapfly's managed-fetch example.","tags":["python","scrapeguide"]},{"path":"/blog/posts/best-financial-data-apis","type":"article","title":"8 Best Financial Data APIs for Developers in 2026","description":"Compare eight financial data APIs for real-time prices, fundamentals, historical data, global markets, technical indicators, and trading apps.","tags":["api","data","python","scrapeguide"]},{"path":"/blog/posts/how-to-bypass-f5-when-web-scraping","type":"article","title":"How to Bypass F5 when Web Scraping in 2026","description":"Bypass F5 when web scraping in 2026: distinguish BIG-IP from Distributed Cloud Bot Defense, inspect its browser telemetry, and run a browser-backed fetch.","tags":["blocking","headless-browser","python"]},{"path":"/blog/posts/best-local-business-data-apis","type":"article","title":"6 Best Local Business Data APIs for Developers in 2026","description":"Compare six local business data APIs for place search, reviews, open map data, categories, and geographic context in 2026.","tags":["api","python"]},{"path":"/blog/posts/best-social-media-data-apis","type":"article","title":"8 Best Social Media Data APIs for Developers in 2026","description":"Which social media API fits your product? Compare eight options by data, auth, limits, and access model, plus when scraping fills the gap in 2026.","tags":["api","data-parsing","python","scrapeguide"]},{"path":"/blog/posts/best-csharp-web-scraping-libraries","type":"article","title":"7 Best C# Web Scraping Libraries in 2026","description":"Compare seven C# web scraping libraries for HTML parsing, JavaScript rendering, browser automation, and site crawling in 2026, with runnable .NET examples.","tags":["data-parsing","frameworks","tools"]},{"path":"/blog/posts/htmlagilitypack-vs-anglesharp","type":"article","title":"HtmlAgilityPack vs AngleSharp: Which C# HTML Parser Should You Use?","description":"Compare Html Agility Pack and AngleSharp for C# HTML parsing: XPath vs CSS selectors, malformed HTML, DOM behavior, speed, and when to use each.","tags":["css-selectors","data-parsing","tools","xpath"]},{"path":"/blog/posts/how-to-scrape-dhl","type":"article","title":"How to Scrape DHL Tracking Status and Shipment Events","description":"Scrape DHL tracking status and shipment events with Python. Inspect /utapi responses, classify outcomes, normalize events, and compare DHL's official API.","tags":["hidden-api","python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-goibibo-hotel-and-flight-prices","type":"article","title":"How to Scrape Goibibo Hotel and Flight Prices","description":"Scrape Goibibo hotel and flight prices with Python in 2026. Use Scrapfly Unblocker (previously ASP), browser rendering, and India sessions to capture structured search data.","tags":["python","scrapeguide"]},{"path":"/blog/posts/best-amazon-scrapers-github","type":"article","title":"4 Best Open-Source Amazon Scrapers in 2026","description":"Four open-source Amazon scraper repos on GitHub in 2026, compared by maintenance, licensing, supported data, and live failure modes.","tags":["data-parsing","python"]},{"path":"/blog/posts/chrome-cdp-stealth-browser-automation-detection-explained","type":"article","title":"Is Chrome CDP Detectable? CDP vs WebDriver for Browser Automation","description":"Is Chrome CDP stealth by default? See how CDP detection works, how CDP differs from WebDriver, and how Playwright, Puppeteer, Selenium, and nodriver compare.","tags":["blocking","cloud-browser","playwright","puppeteer","selenium"]},{"path":"/blog/posts/google-trends-api-alternatives","type":"article","title":"Google Trends API Guide and Alternatives for Developers","description":"Does Google have a Trends API? Yes, in alpha. See its access rules and limits, then compare pytrends, scraping, and managed APIs as alternatives.","tags":["api","hidden-api","python"]},{"path":"/blog/posts/best-real-estate-data-apis","type":"article","title":"6 Best Real Estate Data APIs for Developers in 2026","description":"Which real estate API fits your app? Compare six options for listings, valuations, parcels, zoning, and neighborhood scores in 2026.","tags":["api","python","real-estate"]},{"path":"/blog/posts/best-ecommerce-product-data-apis","type":"article","title":"7 Best E-commerce Product Data APIs for Developers in 2026","description":"Which e-commerce API fits your access? Compare seven options for storefront catalogs, buyer discovery, seller operations, and public product research in 2026.","tags":["api","ecommerce","python"]},{"path":"/blog/posts/scrapy-error-messages","type":"article","title":"Scrapy Error Messages: What They Mean and How to Fix Them","description":"Look up Scrapy error messages by exact string. Find the failing layer and apply the smallest fix without chasing unrelated settings.","tags":["debugging","python","scrapy"]},{"path":"/blog/posts/best-xpath-css-selector-tools","type":"article","title":"7 Best XPath and CSS Selector Tools for Web Scraping in 2026","description":"Compare seven XPath and CSS selector tools for testing selectors, inspecting live DOMs, generating locators, and recording browser flows in 2026.","tags":["data-parsing","tools"]},{"path":"/blog/posts/how-to-scrape-capterra","type":"article","title":"How to Scrape Capterra Reviews and Software Data","description":"Guide to scraping Capterra reviews and software listings with Python. Bypass Cloudflare, extract ratings, pros/cons, reviewer metadata, and paginate review data","tags":["python","scrapeguide"]},{"path":"/blog/posts/best-javascript-web-scraping-libraries","type":"article","title":"Best JavaScript Web Scraping Libraries in 2026 (Node.js Stack Guide)","description":"Compare the 8 best JavaScript and Node.js web scraping libraries in 2026, from axios and Cheerio to Playwright, Puppeteer, and Crawlee, ranked by layer.","tags":["javascript","nodejs","tools"]},{"path":"/blog/posts/best-scrapy-extensions-middlewares","type":"article","title":"9 Scrapy Extensions and Middlewares for 2026","description":"Which Scrapy extensions still work on Scrapy 2.18? 9 packages tested in 2026 for rendering, TLS fingerprints, proxies, and queues, plus the ones that now fail.","tags":["headless-browser","proxies","python","scrapeguide"]},{"path":"/blog/posts/why-scrapy-spider-gets-blocked","type":"article","title":"9 Mechanisms to Check When Your Scrapy Spider Gets Blocked in 2026","description":"Diagnose why a Scrapy spider gets blocked across IP, rate, session, headers, TLS, HTTP/2, JavaScript, behavior, and CAPTCHA signals.","tags":["blocking","python","scrapy"]},{"path":"/blog/posts/best-e-commerce-web-scraping-tools","type":"article","title":"9 Best E-commerce Web Scraping Tools for Developers in 2026","description":"Retrieval, access, extraction, discovery, browser, or screenshots? Pick the narrowest e-commerce scraping tool that fixes your broken layer in 2026.","tags":["ai","api","blocking","headless-browser","python"]},{"path":"/blog/posts/best-web-scraping-tools-for-real-estate","type":"article","title":"6 Best Web Scraping Tools for Real Estate in 2026","description":"Which real estate scraping tool returns the output you are missing? Compare 6 by job in 2026: page HTML, structured JSON, listing URLs, images, parser source.","tags":["python","real-estate","scrapeguide"]},{"path":"/blog/posts/best-open-source-youtube-scrapers","type":"article","title":"6 Best Open-Source YouTube Scrapers in 2026","description":"Compare 6 open-source YouTube scrapers for metadata, transcripts, comments, and search using dated August 2026 maintenance and test evidence.","tags":["python","scrapeguide"]},{"path":"/blog/posts/best-open-source-job-scrapers","type":"article","title":"3 Best Open-Source Job Scrapers for Multi-Board Data in 2026","description":"Compare JobSpy, jobhive, and Levergreen by board coverage, schema, and maintenance, plus Scrapfly's source-available NPOSL-3.0 managed alternative.","tags":["hidden-api","python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-target-com","type":"article","title":"How to Scrape Target.com Product and Pricing Data","description":"Scrape Target.com product and pricing data in 2026 using the Redsky API, tcin and store_id, location-aware prices, and a PerimeterX bypass with Python.","tags":["hidden-api","python","scrapeguide"]},{"path":"/blog/posts/python-web-scraping-projects-from-beginner-to-advanced","type":"article","title":"12 Python Web Scraping Projects for 2026","description":"12 Python web scraping projects laddered by difficulty, from a first BeautifulSoup script to Scrapy crawls, lead databases, and AI agents in 2026.","tags":["ai","data-parsing","headless-browser","python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-skyscanner","type":"article","title":"How to Scrape Skyscanner Flight Prices in Python (2026)","description":"Scrape Skyscanner flight prices in Python. Build result URLs, render the search with the Scrapfly Web Scraping API, and parse the itinerary JSON.","tags":["blocking","headless-browser","python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-airbnb","type":"article","title":"How to Scrape Airbnb Listings and Prices (2026)","description":"Scrape Airbnb listings, prices, reviews, and availability in 2026 with Python. Parse the deferred state JSON, capture the GraphQL XHRs, and bypass blocking.","tags":["python"]},{"path":"/blog/posts/best-linkedin-scrapers-github","type":"article","title":"5 Best Open-Source LinkedIn Scrapers on GitHub in 2026","description":"The 5 best open-source LinkedIn scrapers on GitHub, ranked by auth model and dated maintenance, plus the stale and funnel repos to skip.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-lowe","type":"article","title":"How to Scrape Lowe's Product, Price, Search, and Store Location Data","description":"Scrape Lowe's product, pricing, search, and store-location data in 2026 with Python, embedded JSON, and Scrapfly's Akamai-ready client.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-imdb","type":"article","title":"How to Scrape IMDb Data, Ratings, and Featured Reviews","description":"Fetch protected IMDb pages through Scrapfly Unblocker (previously ASP), then parse JSON-LD and __NEXT_DATA__ for metadata, ratings, featured reviews, search, charts, and people.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-digikey","type":"article","title":"How to Scrape DigiKey Electronic Component Data in 2026","description":"Learn how to scrape DigiKey product data in 2026: parametric search, price breaks, stock, and lifecycle status past Cloudflare, plus when the API fits.","tags":["api","blocking","python","scrapeguide"]},{"path":"/blog/posts/best-open-source-instagram-scrapers","type":"article","title":"5 Best Open-Source Instagram Scrapers in 2026","description":"Compare the 5 best open-source Instagram scrapers in 2026. Ranked by job, auth model, account-ban risk, and maintenance status with working code.","tags":["python","scrapeguide"]},{"path":"/blog/posts/best-mcp-servers-for-web-scraping","type":"article","title":"5 MCP Servers for Web Scraping and Browser Automation in 2026","description":"Which MCP server fits your scraping job? Compare five options for protected sites, browser control, debugging, static fetches, and cross-browser automation.","tags":["ai","tools"]},{"path":"/blog/posts/best-curl-wget-alternatives","type":"article","title":"cURL and Wget Alternatives: 6 Modern Command-Line Tools Compared","description":"6 best curl and wget alternatives in 2026: HTTPie, xh, curlie, wget2, aria2, and Scrapfly CLI, picked by job with a comparison table and working code examples.","tags":["curl","tools","wget"]},{"path":"/blog/posts/best-python-http-clients","type":"article","title":"8 Best Python HTTP Clients for Web Scraping in 2026","description":"Which Python HTTP client should you scrape with? Compare requests, aiohttp, urllib3, niquests, curl_cffi, primp, Stealth-Requests and HTTPX.","tags":["blocking","http","python"]},{"path":"/blog/posts/best-lead-scraping-tools","type":"article","title":"7 Best Lead Scraping Tools in 2026","description":"The 7 best lead scraping tools in 2026, ranked by job. Real scrapers that pull fresh leads from the live web, not rented contact databases, with honest limits.","tags":["ai","api","seo"]},{"path":"/blog/posts/best-browser-fingerprint-testing-tools","type":"article","title":"Browser Fingerprint Testing Tools for Web Scraping in 2026","description":"Test your scraper's TLS, JavaScript, rendering, WebRTC, and DNS fingerprints to isolate the layer exposing it and choose the right fix.","tags":["anti-bot","blocking","tools"]},{"path":"/blog/posts/how-to-scrape-google-jobs","type":"article","title":"How to Scrape Google Jobs in 2026","description":"Learn how to scrape Google Jobs with Python and Scrapfly. Extract job titles, companies, locations, salaries, application links, and descriptions.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-google-play-app-reviews-and-data","type":"article","title":"How to Scrape Google Play App Reviews and Data","description":"Scrape Google Play app reviews and metadata in Python in 2026. Page the full review set past the free library's few-hundred cap without getting throttled.","tags":["hidden-api","python","scrapeguide"]},{"path":"/blog/posts/best-proxy-scrapers","type":"article","title":"4 Best Open-Source Proxy Scrapers and Checkers in 2026","description":"Compare four maintained open-source proxy scrapers and checkers for HTTP, SOCKS4, and SOCKS5 lists, then learn how to validate free proxies in 2026.","tags":["proxies","tools"]},{"path":"/blog/posts/best-ai-browser-agents","type":"article","title":"4 Best AI Browser Agents for Automation and Scraping in 2026","description":"The 4 best AI browser agents in 2026, ranked by what breaks them in production: getting blocked, losing sessions, and failing at scale.","tags":["ai","api","headless-browser"]},{"path":"/blog/posts/how-to-scrape-marriott","type":"article","title":"How to Scrape Marriott Hotel Prices and Availability","description":"How to scrape Marriott hotel prices and room availability with Python in 2026, including search, JSON behind the site, passing Marriott's Akamai bot protection.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-kayak","type":"article","title":"How to Scrape Kayak Flights in 2026","description":"Scrape Kayak flight prices with Python in 2026. Build search URLs, capture the internal flights poll API, and bypass bot detection with Scrapfly Unblocker (previously ASP).","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-rs-online","type":"article","title":"How to Scrape RS-Online (rs-online.com) in 2026","description":"Scrape RS-Online's US listings and product pages in 2026 for prices, stock, specifications, and datasheet links using Python and Scrapfly.","tags":["python","scrapeguide"]},{"path":"/blog/posts/browser-use-vs-playwright","type":"article","title":"Browser Use vs Playwright: Which Should You Use?","description":"Browser Use vs Playwright for scraping: compare architecture, speed, cost, reliability, and failure modes, plus when a hybrid workflow makes sense.","tags":["ai","playwright","python"]},{"path":"/blog/posts/how-to-bypass-aws-waf-when-web-scraping","type":"article","title":"How to Bypass AWS WAF when Web Scraping in 2026","description":"Bypass AWS WAF Bot Control when web scraping in 2026: confirm the 403/202 block, understand how it detects scrapers, and return a clean 200 with Scrapfly Unblocker (previously ASP).","tags":["blocking"]},{"path":"/blog/posts/best-facebook-marketplace-scrapers-github","type":"article","title":"5 Best Open-Source Facebook Marketplace Scrapers on GitHub in 2026","description":"The 5 best open-source Facebook Marketplace scrapers on GitHub, ranked by maintenance and login model, plus the archived repos to avoid.","tags":["python","scrapeguide"]},{"path":"/blog/posts/web-scraping-best-practices","type":"article","title":"Web Scraping Best Practices: 11 Rules for Reliable and Respectful Scraping in 2026","description":"11 web scraping best practices for 2026: robots.txt, rate limiting, hidden APIs, proxy rotation, retries, validation, and monitoring, with working code.","tags":["python","scrapeguide","web-scraping"]},{"path":"/blog/posts/best-websites-to-practice-web-scraping","type":"article","title":"9 Best Websites to Practice Web Scraping in 2026","description":"The 9 best websites to practice web scraping in 2026, ranked and mapped to the skill each teaches, plus the order to drill them and when to move to real sites.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-maersk","type":"article","title":"How to Scrape Maersk Container Tracking and Shipping Schedule Data","description":"Scrape Maersk container tracking events and vessel schedules with Python and Scrapfly. Capture the site's JSON calls after JavaScript rendering.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-emirates-flights","type":"article","title":"How to Scrape Emirates Flights with Python (2026)","description":"Scrape Emirates flight prices, fare brands, schedules, and cabin details in Python with Scrapfly's Web Scraping API. Includes code and anti-bot tips.","tags":["playwright","python"]},{"path":"/blog/posts/human-in-the-loop-cloud-browsers","type":"article","title":"Human-in-the-Loop (HITL) Cloud Browsers","description":"How Human-in-the-Loop cloud browsers hand a live session to a person for CAPTCHAs, MFA, or approvals, then let automation reconnect and finish.","tags":["ai","headless-browser","playwright"]},{"path":"/blog/posts/puppeteer-vs-playwright","type":"article","title":"Puppeteer vs Playwright for Web Scraping","description":"Puppeteer vs Playwright for web scraping in 2026. Compare browser support, auto-waiting, contexts, CDP, speed, and stealth, then pick by workload.","tags":["blocking","headless-browser","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-infinite-scroll-load-more-and-paginated-pages","type":"article","title":"How to Scrape Infinite Scroll, Load More & Paginated Pages","description":"Scrape infinite scroll, load more, and paginated pages in 2026. Spot the pattern, reverse-engineer the API, and fall back to a headless browser correctly.","tags":["headless-browser","hidden-api","python","scrapeguide"]},{"path":"/blog/posts/best-stealth-browsers","type":"article","title":"Best Stealth Browsers for Web Scraping in 2026","description":"Compare the best stealth browsers for web scraping in 2026, including Camoufox, nodriver, SeleniumBase, and Patchright, by speed and maintenance.","tags":["blocking","headless-browser","python","scrapeguide"]},{"path":"/blog/posts/akamai-bot-manager-understanding-abck-cookies-and-sensor-data","type":"article","title":"Akamai Bot Manager: Understanding _abck Cookies and Sensor Data","description":"Understand how Akamai Bot Manager uses _abck cookies, sensor data, TLS, HTTP fingerprints, and JavaScript challenges, with Scrapfly Unblocker (previously ASP) setup.","tags":["blocking","http","javascript","proxies"]},{"path":"/blog/posts/ai-browser-agents-how-to-use-them-with-cloud-browsers","type":"article","title":"AI Browser Agents - How to Use Them with Cloud Browsers","description":"A practical guide to AI browser agents. Learn to integrate Browser Use and Stagehand with cloud browsers for scalable, reliable browser automation AI.","tags":[]},{"path":"/blog/posts/ja3-ja4-tls-fingerprinting-guide-to-detection-and-evasion","type":"article","title":"JA3/JA4 TLS Fingerprinting: Guide to Detection and Evasion","description":"Learn how JA3/JA4 TLS fingerprinting detects web scrapers at the protocol level and how to bypass detection with curl_cffi and Scrapfly.","tags":[]},{"path":"/blog/posts/tls-cipher-suites-in-web-scraping","type":"article","title":"TLS Cipher Suites in Web Scraping","description":"A complete guide to TLS cipher suites for web scraping. Learn how cipher suite fingerprints (JA3/JA4) are used to detect bots and how to avoid blocks.","tags":[]},{"path":"/blog/posts/http-referer-header-complete-guide-for-web-scraping","type":"article","title":"HTTP Referer Header: Complete Guide for Web Scraping","description":"Master the HTTP Referer header for web scraping. This complete guide covers detection mechanics and how to set the Referer correctly to avoid blocks.","tags":[]},{"path":"/blog/posts/web-scraping-with-grok","type":"article","title":"Web Scraping with Grok: Build an Autonomous Scraping Agent","description":"Learn web scraping with Grok in 2026 using Grok Build, grok-cli, the xAI API, and Scrapfly CLI for JavaScript rendering, Unblocker (previously ASP), and structured JSON extraction","tags":[]},{"path":"/blog/posts/how-anti-bot-detection-works","type":"article","title":"How Anti-Bot Detection Works","description":"Learn how anti-bot detection works. We explain the 5 core bot detection techniques, from TLS/JA3 fingerprinting to behavioral analysis.","tags":["blocking"]},{"path":"/blog/posts/how-to-scrape-perplexity","type":"article","title":"How to Scrape Perplexity AI in 2026","description":"Scrape Perplexity AI in Python in 2026. Render JS, bypass Cloudflare with Scrapfly, extract answers and cited domains, and build a citation tracker.","tags":["ai","hidden-api","python","scrapeguide"]},{"path":"/blog/posts/web-scraping-with-hermes-agent","type":"article","title":"Web Scraping with Hermes Agent","description":"Learn why Hermes Agent's built-in scraping breaks on protected sites and how to add the Scrapfly skill or MCP server for anti-bot bypass at scale.","tags":["ai","api","blocking","scrapeguide"]},{"path":"/blog/posts/what-is-an-anti-detect-browser","type":"article","title":"What Is an Anti-Detect Browser? How They Work and When to Use One","description":"What is an anti-detect browser, fingerprint spoofing and profile isolation. how anti-detect browsers compare to headless and cloud browsers, and when to use it.","tags":["anti-bot","blocking"]},{"path":"/blog/posts/best-browser-automation-tools","type":"article","title":"Best Browser Automation Tools in 2026","description":"Cloud or self-hosted? Compare browser automation tools by protocol, protected-site support, session state, and failure evidence in 2026.","tags":["api","headless-browser","playwright"]},{"path":"/blog/posts/how-to-scrape-google-flights","type":"article","title":"How to Scrape Google Flights Data in 2026","description":"Learn how to scrape Google Flights with Python and Scrapfly: extract prices, airlines, routes, stops, layovers, and fare data from rendered search results.","tags":["python","scrapeguide"]},{"path":"/blog/posts/best-open-source-web-scrapers","type":"article","title":"10 Best Open-Source Web Scrapers in 2026","description":"The 10 best open-source web scrapers in 2026 ranked by maintenance, license, and anti-bot ceiling. Covers Scrapy, Crawlee, Playwright, Camoufox, Crawl4AI, Colly","tags":["crawling","javascript","python"]},{"path":"/blog/posts/curl-with-proxy","type":"article","title":"How to Use cURL With a Proxy in 2026","description":"Use cURL with a proxy the right way. Cover the -x flag, http_proxy and ALL_PROXY env vars, .curlrc, SOCKS5 vs socks5h, -U vs -u auth, and proxy error fixes.","tags":["curl","http","proxies"]},{"path":"/blog/posts/how-to-build-a-product-data-pipeline-from-multiple-ecommerce-sites","type":"article","title":"How to Build a Product Scraper for Multiple Sites","description":"Build a multi-source product scraping pipeline with canonical schema design, config-driven adapters, data normalisation, and product matching in Python.","tags":["ecommerce","project","python"]},{"path":"/blog/posts/how-to-avoid-proxy-detection","type":"article","title":"How to Avoid Proxy Detection in Web Scraping","description":"How to avoid proxy detection in web scraping: IP trust signals, residential vs datacenter proxies, and rotation vs sticky session strategy.","tags":["blocking","proxies","scrapeguide"]},{"path":"/blog/posts/scraping-government-public-records-data","type":"article","title":"How to Scrape Government and Public Records Data","description":"Learn how to scrape government and public records data in 2026: sources, anti-bot friction, hidden APIs, and normalizing records across 50 states.","tags":["data-parsing","hidden-api","python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-air-france-flights","type":"article","title":"How to Scrape Air France Flights with Python in 2026","description":"Scrape Air France flights with Python in 2026 by capturing the GraphQL booking response: fares, routes, layovers, aircraft, CO2, and blocking.","tags":["api","blocking","headless-browser","python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-leads","type":"article","title":"Web Scraping for Lead Generation: Build Your Own B2B Database","description":"Engineer's guide to building a B2B lead database with web scraping: source discovery, Scrapfly extraction, dedup, enrichment, and CRM delivery.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-build-a-grocery-price-comparison-tool-with-python","type":"article","title":"How to Build a Grocery Price Comparison Tool with Python","description":"Learn how to build a grocery price comparison tool with Python that scrapes location-aware prices from Walmart, Instacart, and Kroger and compares them.","tags":["ecommerce","project","python"]},{"path":"/blog/posts/ai-agent-web-scraping","type":"article","title":"Web Scraping for AI Agents in 2026","description":"A builder's guide to AI agent web scraping: four building blocks, four agent shapes, and the Scrapfly surfaces that match each one.","tags":["ai","api","python","scrapeguide"]},{"path":"/blog/posts/best-social-media-scraping-tools","type":"article","title":"The Best Open-Source Social Media Scrapers for 2026","description":"Compare social media scraping tools in 2026. Free open-source scrapers for Instagram, X, LinkedIn, TikTok, YouTube, Reddit, Threads, and Facebook with code.","tags":["python","web-scraping"]},{"path":"/blog/posts/best-anti-bot-bypass-tools","type":"article","title":"11 Best Anti-Bot Bypass Tools for Web Scraping in 2026","description":"Compare 11 anti-bot bypass tools for web scraping in 2026. Learn when to use HTTP, a rendering API, or stateful browser automation.","tags":["blocking","tools","web-scraping"]},{"path":"/blog/posts/best-tools-for-ai-webscraping","type":"article","title":"Best AI Web Scraping Tools for LLM and RAG Pipelines in 2026","description":"The best AI web scraping tools in 2026, ranked by job: prompt extraction, agentic browsing, MCP servers, and open-source crawlers that survive anti-bot.","tags":[]},{"path":"/blog/posts/best-public-data-sources-for-lead-generation","type":"article","title":"10 Best Public Data Sources for Lead Generation in 2026","description":"10 best public data sources for lead generation in 2026, ranked by value with the fields, access method (API, bulk, or scrape), and freshness of each.","tags":["api","data-parsing"]},{"path":"/blog/posts/how-to-scrape-large-product-catalogs","type":"article","title":"How to Scrape an Entire Product Catalogue with Python","description":"Scrape a full product catalog in Python: discover every product URL, paginate to exhaustion, extract clean data, and re-crawl for changes.","tags":["data-parsing","python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-chatgpt","type":"article","title":"How to Scrape ChatGPT Responses in 2026","description":"Learn how to scrape ChatGPT responses with Python in 2026, handle Cloudflare and streaming responses, and persist login with Cloud Browser session resume.","tags":["ai","python"]},{"path":"/blog/posts/how-to-build-a-web-scraping-agent-with-claude","type":"article","title":"How to Build a Web Scraping Agent with Claude","description":"Build a Claude web scraping agent that handles JavaScript rendering, Cloudflare, and anti-bot protection, with Scrapfly CLI, Claude Code skills, and Python API examples.","tags":["ai","python"]},{"path":"/blog/posts/gemini-for-webscraping","type":"article","title":"How to Build a Web Scraping Agent with Gemini","description":"Build a Gemini web scraping agent on real protected sites. Gemini CLI skills, URL Context limits, Python pipelines, and anti-bot bypass with Scrapfly.","tags":["ai","api","python","scrapeguide"]},{"path":"/blog/posts/how-to-bypass-anti-bot-protection","type":"article","title":"How to Bypass Anti-Bot Protection in 2026: All 8 Major Vendors","description":"Learn to identify and bypass all major anti-bot systems: Cloudflare, DataDome, PerimeterX, Kasada, Akamai, Incapsula, F5, and AWS WAF. Python code included.","tags":["blocking","python","scrapeguide"]},{"path":"/blog/posts/http2-http3-fingerprinting-guide","type":"article","title":"HTTP/2 and HTTP/3 Fingerprinting: Protocol-Level Bot Detection","description":"Complete guide to HTTP/2 and HTTP/3 fingerprinting for web scraping. Learn how protocol-level bot detection works and how to bypass it with Python.","tags":["blocking","http","python","tools"]},{"path":"/blog/posts/how-browser-fingerprinting-works","type":"article","title":"How Browser Fingerprinting Works and How to Defend Against It","description":"A deep dive into browser fingerprinting for developers. Learn how canvas, WebGL, and WebGPU fingerprinting work, test your setup, and bypass anti-bot detection.","tags":[]},{"path":"/blog/posts/browser-automation-api-use-cases-patterns","type":"article","title":"Browser Automation API Use Cases and Patterns","description":"Learn what browser automation APIs do, how cloud browser architecture works, and how to use them for scraping, testing, and AI agents.","tags":["api","headless-browser","playwright"]},{"path":"/blog/posts/visual-ecommerce-price-monitoring-with-automated-screenshots","type":"article","title":"Visual Ecommerce Price Monitoring with Automated Screenshots","description":"Monitor ecommerce prices with automated screenshots, HTML extraction, and OCR fallback, then keep visual evidence for every detected price change.","tags":["ecommerce","python","screenshots"]},{"path":"/blog/posts/tcp-ip-stack-fingerprinting-proxy-bypass","type":"article","title":"TCP/IP Stack Fingerprinting and Proxy Bypass","description":"How TCP/IP stack fingerprinting detects proxies and scrapers from SYN packets, why proxies leak a Linux signature, and how to test and bypass it.","tags":["blocking","proxies","scrapeguide"]},{"path":"/blog/posts/playwright-stealth-bypass-bot-detection","type":"article","title":"Playwright Stealth: Bypass Bot Detection in Python & Node.js","description":"Learn how to use Playwright stealth to bypass bot detection in Python and Node.js. Covers stealth setup, evasion modules, and detection testing.","tags":["blocking","headless-browser","nodejs","python"]},{"path":"/blog/posts/how-cloudflare-detects-bots","type":"article","title":"How Cloudflare Detects Bots: TLS, HTTP/2, Canvas, and Turnstile Explained","description":"How Cloudflare's 7+ bot detection layers work, including TLS fingerprinting, HTTP/2, Canvas, WebGL, and how to scrape protected sites.","tags":["blocking","python"]},{"path":"/blog/posts/stagehand-vs-browser-use","type":"article","title":"Stagehand vs Browser Use: AI Browser Agent Guide","description":"Stagehand vs Browser Use comparison: features, LLM support, API design, and when to choose each AI browser automation framework.","tags":["cloud-browser","python","typescript"]},{"path":"/blog/posts/post-quantum-tls-bot-detection","type":"article","title":"Post-Quantum TLS: Why Scraping Tools Are Now Exposed","description":"How post-quantum TLS creates new bot detection signals in 2026, the uTLS CVE vulnerabilities, CDN enforcement by Cloudflare and Akamai, and what to do now.","tags":["blocking","http"]},{"path":"/blog/posts/how-to-bypass-anti-bot-protection-when-web-scraping","type":"article","title":"How to Bypass Anti-Bot Protection When Web Scraping","description":"Guide to bypassing anti-bot protection when web scraping. Covers detection methods, 5 universal bypass techniques, and system-specific strategies.","tags":["blocking"]},{"path":"/blog/posts/how-to-bypass-cloudflare-turnstile","type":"article","title":"How to Bypass Cloudflare Turnstile","description":"Guide to bypassing Cloudflare Turnstile CAPTCHA when web scraping. Covers detection methods, browser automation, CAPTCHA solvers, and cloud browsers.","tags":["blocking"]},{"path":"/blog/posts/openclaw-cloud-browser-social-media-automation","type":"article","title":"How to Automate Social Media Tasks with OpenClaw and Cloud Browsers","description":"Learn to configure OpenClaw with Scrapfly cloud browsers for social media automation. Bypass anti-bot detection on Twitter, Instagram, and LinkedIn.","tags":["ai","headless-browser"]},{"path":"/blog/posts/puppeteer-stealth-complete-guide","type":"article","title":"Puppeteer Stealth: Complete Guide to Avoiding Detection","description":"Learn how to use Puppeteer Stealth plugin to avoid detection. Complete tutorial covering stealth configuration, evasion modules, proxy rotation, and scaling solutions.","tags":["blocking","headless-browser","nodejs","puppeteer"]},{"path":"/blog/posts/how-to-create-an-ai-browser-agent-for-free","type":"article","title":"How to Create an AI Browser Agent for Free","description":"Create free AI browser agents with Browser-Use and Stagehand. Working code examples using Gemini's free tier for browser automation.","tags":["ai","headless-browser","python","typescript"]},{"path":"/blog/posts/headless-browser-vs-cloud-browser","type":"article","title":"Headless Browser vs Cloud Browser: Which One Should You Choose?","description":"Comprehensive comparison of headless browsers vs cloud browsers for web scraping and automation. Understand costs, scaling, and when to use each approach.","tags":[]},{"path":"/blog/posts/best-captcha-solving-api","type":"article","title":"6 Best CAPTCHA Solving APIs for Web Scraping in 2026","description":"Compare 6 CAPTCHA solving APIs for reCAPTCHA, Turnstile, and browser automation. See current pricing, integration models, and safe selection criteria.","tags":["blocking"]},{"path":"/blog/posts/what-is-a-cloud-browser","type":"article","title":"Cloud Browser: What Is It and How to Use It","description":"Understand cloud browsers: what they are, how they work, and how to use them for web scraping, testing, and AI browser agents with practical code examples.","tags":["headless-browser"]},{"path":"/blog/posts/best-cloud-browser-apis","type":"article","title":"7 Best Cloud Browser APIs for Web Scraping in 2026","description":"Compare the best cloud and headless browser APIs for web scraping in 2026. Managed stealth browsers and open-source engines, ranked by the job they do best.","tags":["ai","api","headless-browser"]},{"path":"/blog/posts/what-is-a-browser-as-a-service","type":"article","title":"What Is Browser as a Service?","description":"Learn what Browser as a Service is and why managed cloud browsers beat DIY approaches for web scraping, testing, and AI agents.","tags":["ai","cloud-browser","tools"]},{"path":"/blog/posts/what-is-screen-scraping","type":"article","title":"What is Screen Scraping? Developer's Guide with Python Examples","description":"Learn screen scraping with Python, Playwright, and Selenium. Extract data from JavaScript websites with code examples and best practices.","tags":["playwright","python","selenium"]},{"path":"/blog/posts/how-to-take-screenshots-php","type":"article","title":"How to Take Screenshots in PHP?","description":"Learn how to take screenshots in PHP using Browsershot, Chrome-PHP, and APIs. Covers installation, wait strategies, device emulation, and advanced techniques.","tags":["headless-browser","php","screenshots"]},{"path":"/blog/posts/competitor-price-monitoring-with-crawler-api","type":"article","title":"Competitor Price Monitoring with Crawler API","description":"Complete guide to building a competitor price monitoring system with Scrapfly Crawler API. Handle anti-bot protection on e-commerce sites and automate repricing workflows.","tags":["crawling","ecommerce","python"]},{"path":"/blog/posts/how-to-screenshot-cloudflare-protected-websites","type":"article","title":"How to Screenshot Cloudflare-Protected Websites","description":"Complete guide to screenshotting Cloudflare-protected websites with Scrapfly Screenshot API, Puppeteer, and Selenium. Bypass anti-bot detection with code examples.","tags":["blocking","python","screenshots"]},{"path":"/blog/posts/build-a-documentation-chatbot-that-works-on-any-website","type":"article","title":"Build a Documentation Chatbot That Works on ANY Website","description":"Build a documentation chatbot with Python using Scrapfly, LangChain, and Streamlit. Works on protected sites like Cloudflare Docs.","tags":["ai","crawling","python"]},{"path":"/blog/posts/scraper-api-vs-crawler-api","type":"article","title":"Scraper API vs Crawler API - When to Use Each for AI","description":"Compare Scraper API vs Crawler API for AI data collection. Code examples, anti-bot config, and RAG recommendations for both Scrapfly APIs.","tags":["ai","api","tools"]},{"path":"/blog/posts/langchain-web-scraping-complete-guide-scrapfly","type":"article","title":"LangChain Web Scraping: Build AI Agents & RAG Applications","description":"Complete guide to using LangChain for web scraping with Scrapfly. Build AI agents, RAG applications, and smart data extraction pipelines with Python examples.","tags":["ai","python"]},{"path":"/blog/posts/best-web-scraping-apis","type":"article","title":"11 Best Web Scraping APIs, Libraries, and Crawlers for Developers in 2026","description":"Compare the best web scraping APIs, parsers, browser tools, and crawlers for production scraping. See where Scrapfly, BeautifulSoup, Playwright, Scrapy, and more fit in the stack.","tags":["nodejs","python"]},{"path":"/blog/posts/how-to-scrape-facebook","type":"article","title":"How to Scrape Facebook Marketplace and Events With Python","description":"Scrape public Facebook Pages, Posts, Marketplace, Events, and Groups in Python with Scrapfly: anti-bot bypass, residential proxies, and JS rendering.","tags":["playwright","python","scrapeguide"]},{"path":"/blog/posts/web-scraping-services-explained","type":"article","title":"Web Scraping Services Explained","description":"Complete guide to web scraping services covering managed APIs, self-hosted platforms, pricing models, and best practices for data extraction.","tags":["tools"]},{"path":"/blog/posts/social-media-scraping","type":"article","title":"Social Media Scraping in 2026","description":"Complete guide to scraping Instagram, Twitter, TikTok, and LinkedIn with Python. Learn anti-blocking techniques for 2026.","tags":["python","scrapeguide"]},{"path":"/blog/posts/top-langchain-alternatives","type":"article","title":"Top LangChain Alternatives in 2026: Which One Should You Choose?","description":"Explore the top LangChain alternatives in 2026, including LlamaIndex, Haystack, AutoGPT, and more. Compare features, performance, and use cases.","tags":["ai"]},{"path":"/blog/posts/how-to-scrape-naver","type":"article","title":"How to Scrape Naver.com: Search, Images, News & Blog Posts (Update 2026)","description":"Scrape Naver.com search, images, news, blog, shopping, and finance with Python. Developers API, entry.bootstrap() JSON, and hidden endpoints.","tags":["beautifulsoup","python","requests","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-imovelweb","type":"article","title":"How to Scrape Imovelweb Without Getting Blocked in 2026","description":"Scrape Imovelweb property listings with Python and BeautifulSoup. Bypass DataDome and geolocation using Scrapfly proxies.","tags":["beautifulsoup","python","requests","scrapeguide","tools"]},{"path":"/blog/posts/how-to-scrape-autoscout24","type":"article","title":"How to Scrape AutoScout24: API, Anti-Bot Bypass, and Python Guide","description":"Learn how to scrape AutoScout24 with Python, handle Akamai blocking, understand the official API limits, and extract car listings, prices, and specs.","tags":["python","requests","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-allegro","type":"article","title":"How to Scrape Allegro.pl in 2026","description":"Bypass DataDome and scrape Allegro.pl with Python. Extract product listings, prices, specs, and seller data using itemprop meta tags and Polish proxies.","tags":["beautifulsoup","python","requests","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-ticketmaster","type":"article","title":"How to Scrape Ticketmaster in 2026","description":"Scrape public Ticketmaster event listings, artist tour dates, venues, and price ranges in Python. Anti-bot bypass, JSON-LD parsing, Discovery API alternatives.","tags":["beautifulsoup","python","requests","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-mouser-com-in-2026","type":"article","title":"How to Scrape Mouser.com in 2026","description":"Learn how to scrape Mouser.com in 2026. Bypass Akamai Bot Manager, handle region-based pricing, and pull product data from JSON-LD and search results.","tags":["beautifulsoup","python","requests","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-zoro-dot-com","type":"article","title":"How to Scrape Zoro.com Without Getting Blocked in 2026","description":"Scrape Zoro.com product titles, prices, SKUs, and attributes with Python. Bypass DataDome anti-bot protection using stable data-za analytics selectors and BeautifulSoup","tags":["beautifulsoup","python","requests","scrapeguide"]},{"path":"/blog/posts/how-to-stop-wasting-money-on-proxies","type":"article","title":"How to Stop Wasting Money on Proxies","description":"Discover how developers can cut proxy costs by optimizing traffic, choosing the right providers, and leveraging Scrapfly Proxy Saver for efficient web scraping.","tags":["proxies"]},{"path":"/blog/posts/https-vs-socks-proxies","type":"article","title":"SOCKS5 vs HTTP Proxy: Key Differences and When to Use Each","description":"Compare HTTP CONNECT and SOCKS5 proxies for scraping: tunnels, DNS resolution, authentication, UDP and QUIC limits, plus tested Python and cURL setup.","tags":["http","proxies"]},{"path":"/blog/posts/optimize-proxy-bandwidth-with-image-and-css-stubbing","type":"article","title":"Optimize Proxy Bandwidth with Image & CSS Stubbing","description":"Reduce proxy costs by 30-50% through intelligent image and CSS stubbing techniques that eliminate unnecessary resource downloads while preserving functionality.","tags":["proxies"]},{"path":"/blog/posts/what-is-a-reverse-proxy","type":"article","title":"What is a Reverse Proxy?","description":"Understand reverse proxies, their differences from forward proxies, and how they enable load balancing, security and caching in modern web infrastructure.","tags":["proxies"]},{"path":"/blog/posts/bypass-proxy-detection-with-browser-fingerprint-impersonation","type":"article","title":"Bypass Proxy Detection with Browser Fingerprint Impersonation","description":"Stop proxy blocks with browser fingerprint impersonation using this guide for Playwright, Selenium, curl-impersonate & Scrapfly","tags":["blocking","playwright","proxies","puppeteer","selenium"]},{"path":"/blog/posts/how-to-scrape-youtube","type":"article","title":"How to Scrape YouTube in 2026","description":"Learn how to scrape YouTube, channel, video, and comment data using Python using direct JSON endpoints. Step-by-step youtube scraping guide with code examples.","tags":["hidden-api","python","scrapeguide"]},{"path":"/blog/posts/how-caching-can-cut-your-proxy-bill","type":"article","title":"How Caching Can Cut Your Proxy Bill by 70%","description":"Learn how intelligent caching strategies can reduce proxy costs by 40-70%. Complete guide to bandwidth optimization and proxy management.","tags":["proxies"]},{"path":"/blog/posts/how-to-optimize-netnut-proxies","type":"article","title":"How to Optimize NetNut Proxies","description":"Learn how to set up and optimize NetNut proxies for web scraping, including bandwidth reduction techniques and integration with Scrapfly Proxy Saver.","tags":["proxies"]},{"path":"/blog/posts/how-to-optimize-oxylabs-proxies","type":"article","title":"How to Optimize Oxylabs Proxies","description":"Learn how to optimize Oxylabs proxies for efficient web scraping using Python and Scrapfly Proxy Saver. Reduce bandwidth, improve speed, and cut costs.","tags":["proxies"]},{"path":"/blog/posts/how-to-reduce-your-bright-data-bandwidth-usage","type":"article","title":"How to Reduce Your Bright Data Bandwidth Usage","description":"Learn how to reduce Bright Data proxy bandwidth usage using Python optimizations and Scrapfly Proxy Saver to cut data costs by up to 30%","tags":["proxies"]},{"path":"/blog/posts/what-is-rate-limiting-everything-you-need-to-know","type":"article","title":"What is Rate Limiting? Everything You Need to Know","description":"Discover what rate limiting is, why it matters, how it works, and how developers can implement it to build stable, scalable applications.","tags":["blocking","crawling","http"]},{"path":"/blog/posts/how-to-optimize-proxies","type":"article","title":"How to Optimize Proxies","description":"Learn how to optimize proxies for speed, anonymity, and cost. Includes comparisons of proxy vs VPN, and tips for developers using Scrapfly.","tags":["proxies"]},{"path":"/blog/posts/how-to-build-an-mcp-server","type":"article","title":"How to Build an MCP Server in Python","description":"Build an MCP server in Python with tools, resources, and prompts. A beginner's guide to the model context protocol using a simple calculator example.","tags":["ai"]},{"path":"/blog/posts/what-is-mcp-understanding-the-model-context-protocol","type":"article","title":"What Is MCP? Understanding the Model Context Protocol","description":"What is MCP? Learn how the Model Context Protocol powers tools like Copilot Studio by giving AI models access to real-time, structured context.","tags":["ai"]},{"path":"/blog/posts/build-a-proxy-api-rotate-proxies-and-save-bandwidth","type":"article","title":"How to Build a Proxy Rotation API With mitmproxy","description":"Build a rotating proxy API with mitmproxy. Add sticky sessions, health scoring, client-owned retries, and safe caching, then compare managed and BYOP routes.","tags":["api","proxies"]},{"path":"/blog/posts/what-is-http-407-status-code-and-how-to-fix-it","type":"article","title":"What is HTTP 407 Status Code and How to Fix it","description":"Imagine trying to access a website, only to be stopped in your tracks by a frustrating error code. If you've encountered HTTP 407, you know how disruptive it...","tags":["http"]},{"path":"/blog/posts/what-is-http-405-error","type":"article","title":"What is HTTP 405 Error?  (Method Not Allowed)","description":"HTTP error codes can be confusing, especially when they disrupt your web scraping or automation tasks. One such error is the HTTP 405 error, which signals a...","tags":[]},{"path":"/blog/posts/what-is-a-proxy-server","type":"article","title":"What Is a Proxy Server?","description":"A proxy server is one of those technologies every developer has heard of, but few truly understand beyond the basics of \"it hides my IP address.\" In reality...","tags":[]},{"path":"/blog/posts/what-is-499-status-code-client-closed-request","type":"article","title":"What is HTTP 499 Status Code and How to Fix it?","description":"Imagine this: You’re surfing the web or managing your server when, suddenly, you’re greeted with an error you’ve never encountered before: 499. It’s not as...","tags":["http"]},{"path":"/blog/posts/top-web-crawler-tools","type":"article","title":"Top Web Crawler Tools in 2026","description":"Web crawling has evolved dramatically in 2026. What used to require complex infrastructure and constant maintenance can now be accomplished with a few clicks...","tags":["tools"]},{"path":"/blog/posts/top-5-residential-proxy-providers","type":"article","title":"5 Best Residential Proxy Providers for Web Scraping in 2026","description":"Compare six residential proxy providers by public entry path, geo targeting, and session controls, then choose raw inventory, managed fetching, or BYOP optimization in 2026.","tags":["blocking","proxies"]},{"path":"/blog/posts/the-best-datacenter-proxies","type":"article","title":"6 Best Datacenter Proxy Providers for Web Scraping in 2026","description":"Compare 6 datacenter proxy providers for web scraping in 2026 by inventory mode, billing unit, geography, and fair-use limits, using August 2026 product facts.","tags":["proxies"]},{"path":"/blog/posts/how-to-track-web-page-changes-using-automated-screenshots","type":"article","title":"How to Track Web Page Changes with Automated Screenshots","description":"There are many different ways to monitor web page changes and one of the most popular techniques is screenshot tracking.\nWith this method only the final visual...","tags":["python","screenshots"]},{"path":"/blog/posts/how-to-retry-in-axios","type":"article","title":"How to Retry in Axios","description":"Connection errors are a common occurrence in HTTP requests. Especially in web scraping where servers may impose strict rate limits or block repeated requests...","tags":[]},{"path":"/blog/posts/how-to-optimize-webshare-proxies","type":"article","title":"How to Optimize Webshare Proxies","description":"Websharehttps://www.webshare.io/ is a fast-growing proxy provider offering affordable, reliable proxy solutions for various web scraping and automation tasks...","tags":[]},{"path":"/blog/posts/how-to-make-instant-data-scraper","type":"article","title":"Instant Data Scraper Guide - Web Scraping with No Code","description":"Web scraping has become an essential tool for accessing and organizing data from the web, but coding skills are often seen as a barrier. Thankfully, no-code...","tags":["tools"]},{"path":"/blog/posts/how-to-curl-download-file","type":"article","title":"How to Use cURL to Download Files","description":"Curlhttps://curl.se/, short for \"Client URL,\" is a versatile command-line tool used for transferring data with URLs. It's widely favored by developers and...","tags":[]},{"path":"/blog/posts/guide-to-yahoo-finance-api","type":"article","title":"Yahoo Finance API: How to Get Yahoo Finance Data in Python (2026)","description":"How to get Yahoo Finance data in Python: the yfinance library, the hidden chart endpoint, scraping, rate limits, and reliable data at scale.","tags":["api","data-parsing","hidden-api","python","scrapeguide"]},{"path":"/blog/posts/guide-to-google-news-api-and-alternatives","type":"article","title":"Guide to Google News API and Alternatives","description":"In a world of endless information, accessing news data efficiently can be vital for many businesses. Google News has been a trusted news aggregator for a...","tags":["api"]},{"path":"/blog/posts/curl-vs-wget","type":"article","title":"cURL vs Wget: Key Differences Explained","description":"In the world of web command-line tools, two names frequently come up: cURL and wget. Whether you're a web developer, system administrator or just want to...","tags":[]},{"path":"/blog/posts/css-selector-cheatsheet","type":"article","title":"Ultimate CSS Selector Cheatsheet for Web Scraping and HTML Parsing","description":"CSS selectors is a powerful HTML querying protocol which is used by browsers to determine what HTML elements to style.\nIt's also incredibly useful in HTML...","tags":["css-selectors","data-parsing"]},{"path":"/blog/posts/crawl4AI-explained","type":"article","title":"Crawl4AI Guide: Web Crawling for LLMs, RAG, and AI Agents","description":"Learn Crawl4AI v0.8.x with working Python examples. Covers LLM extraction, deep crawling, adaptive crawling, anti-bot bypass, and Docker deployment","tags":[]},{"path":"/blog/posts/crawler-apis-for-monitoring-website-changes-maintaining-ai-chatbots","type":"article","title":"Crawler APIs for Monitoring Website Changes","description":"If your AI chatbot doesn't reflect what's changed on the web, users notice.\nFrom pricing updates to policy changes and product docs, websites evolve constantly...","tags":[]},{"path":"/blog/posts/best-proxy-providers-for-web-scraping","type":"article","title":"Best Proxy Providers for Web Scraping: Choose by Type (2026)","description":"Compare datacenter, residential, ISP, and mobile proxies by decision gate, then choose direct inventory, managed fetching, or an egress optimizer.","tags":["proxies"]},{"path":"/blog/posts/advanced-proxy-connection-optimization-techniques","type":"article","title":"Advanced Proxy Connection Optimization Techniques","description":"Proxy performance bottlenecks often stem from inefficient connection management rather than bandwidth limitations or server capacity. While most organizations...","tags":[]},{"path":"/blog/posts/automatic-failover-strategies-for-reliable-data-extraction","type":"article","title":"Automatic Failover Strategies for Reliable Data Extraction","description":"In the world of web scraping, failure is not an \"if\" but a \"when\". Network connections drop, websites deploy new anti-bot measures, servers return unexpected...","tags":["blocking"]},{"path":"/blog/posts/gpt-crawler-a-complete-guide-to-automated-web-data-collection-for-ai-training","type":"article","title":"GPT Crawler: The AI Training Data Collection Guide","description":"Learn how to use GPT Crawler to collect web data for AI training. A developer's guide with setup tips, configuration steps, and best practices.","tags":["ai","crawling"]},{"path":"/blog/posts/how-to-choose-the-best-proxy-unblocker","type":"article","title":"How to Choose a Web Unblocker for Web Scraping (and When a Proxy Is Enough)","description":"When does a proxy stop being enough for web scraping? A 2026 guide to picking proxies, web unblockers, or scraping browsers without overpaying.","tags":["blocking","proxies"]},{"path":"/blog/posts/guide-to-google-image-search-api-and-alternatives","type":"article","title":"Guide To Google Image Search API and Alternatives","description":"Learn about Google Image Search API alternatives, including Bing API and scraping techniques. Implement image search functionality in your applications.","tags":["api"]},{"path":"/blog/posts/guide-to-list-crawling","type":"article","title":"Guide to List Crawling: Everything You Need to Know","description":"List crawling tutorial: bypass anti-bot systems, assess defenses, choose tools (BeautifulSoup vs Playwright), code examples, troubleshooting.","tags":["beautifulsoup","crawling","python"]},{"path":"/blog/posts/google-scholar-api-and-alternatives","type":"article","title":"Guide to Google Scholar API and Alternatives","description":"Learn how to access Google Scholar data without an official API. Explore alternatives and the best methods for scientific data automation.","tags":["api"]},{"path":"/blog/posts/how-to-curl-json","type":"article","title":"How to POST JSON With cURL in 2026: Inline, File, and jq","description":"POST JSON with cURL using --json, files, or stdin. Add Bearer auth, parse responses with jq, and fix curl JSON 400, 401, and 415 errors.","tags":["curl"]},{"path":"/blog/posts/google-serp-api-and-alternatives","type":"article","title":"Best SERP APIs in 2026: Official Google Alternatives & Third-Party Providers","description":"No official Google SERP API exists. Compare DuckDuckGo, Brave, Yandex and APIs (SerpApi, Serper, DataForSEO, Scrapfly) with benchmarks and pricing.","tags":["seo"]},{"path":"/blog/posts/proxy-vs-vpn","type":"article","title":"Proxy vs VPN: In-Depth Comparison","description":"Explore the proxy vs vpn debate with insights on key differences, benefits, limitations and alternatives. Discover when to choose a proxy or VPN.","tags":["proxies"]},{"path":"/blog/posts/how-to-automate-chrome-screenshots","type":"article","title":"How to Automate Website Screenshots with Python & JavaScript","description":"Learn how to automate Chrome screenshots with Playwright, Selenium, Puppeteer, browser commands, extensions, and APIs for efficient workflows.","tags":["screenshots"]},{"path":"/blog/posts/guide-to-llm-training-fine-tuning-and-rag","type":"article","title":"Guide to LLM Training, Fine-Tuning, and RAG","description":"Differences between LLM training, fine-tuning, and RAG. Learn how to use pre-trained models for custom tasks and real-time knowledge retrieval.","tags":["ai"]},{"path":"/blog/posts/practical-guide-to-llm-agents","type":"article","title":"Guide to Understanding and Developing LLM Agents","description":"Learn about LLM agents: what are they, the components they are made of and real life example of how to develop your own llm agent with langchain","tags":["ai"]},{"path":"/blog/posts/guide-to-google-jobs-api-and-alternatives","type":"article","title":"Guide to Google Jobs API and Alternatives","description":"Explore Google Jobs API alternatives like structured data, web scraping, and third-party job APIs to integrate job listings.","tags":["api"]},{"path":"/blog/posts/how-to-find-all-urls-on-a-domain","type":"article","title":"How to Find All URLs on a Domain","description":"Learn how to efficiently find all URLs on a domain using Python and web crawling. Guide on how to crawl entire domain to collect all website data","tags":["crawling","python"]},{"path":"/blog/posts/what-are-googlebot-user-agent-strings","type":"article","title":"What is Googlebot User Agent String?","description":"Learn about Googlebot user agents, how to verify them, block unwanted crawlers, and optimize your site for better indexing and SEO performance.","tags":["crawling","seo"]},{"path":"/blog/posts/what-is-cloudscraper-and-new-alternatives","type":"article","title":"Alternatives to Cloudscraper to Bypass Cloudflare","description":"Learn why Cloudscraper is outdated and explore modern alternatives for bypassing Cloudflare protections effectively and ethically.","tags":["blocking"]},{"path":"/blog/posts/screenshot-to-pdf","type":"article","title":"How to Capture and Convert a Screenshot to PDF","description":"Convert screenshots to PDF with Python and Node.js using tools like Pillow, pdfkit, and Puppeteer for easy documentation and professional reports","tags":["nodejs","python","screenshots"]},{"path":"/blog/posts/playwright-examples-javascript-and-python","type":"article","title":"Playwright Examples for Web Scraping and Automation","description":"Learn Playwright with Python and JavaScript (nodejs, bun, deno) examples for automating browsers like Chromium, WebKit, and Firefox and scraping","tags":["nodejs","playwright","python"]},{"path":"/blog/posts/web-scraping-with-playwright-and-javascript","type":"article","title":"Web Scraping with Playwright and JavaScript","description":"Learn about web scraping using Playwright - a browser automation library for server side JavaScript like NodeJS, Deno or Bun. Plus example project","tags":["headless-browser","nodejs","playwright"]},{"path":"/blog/posts/python-wget-guide","type":"article","title":"How to use wget in Python","description":"Learn how to use wget in Python through subprocess calls and the best wget alternatives to downloading files using python.","tags":["python"]},{"path":"/blog/posts/how-to-use-python-to-parse-json","type":"article","title":"Ultimate Guide to JSON Parsing in Python","description":"Learn JSON parsing in Python with this ultimate guide. Explore basic and advanced techniques using json, and tools like ijson and nested-lookup.","tags":["data-parsing","python"]},{"path":"/blog/posts/guide-to-javascript-axios-headers","type":"article","title":"How to Set Axios Headers: Complete Guide with Examples (2026)","description":"Complete guide to Axios headers: per-request config, global defaults, instances, interceptors, header merging precedence, and web scraping patterns.","tags":["http","javascript","nodejs"]},{"path":"/blog/posts/guide-to-html-parsing-with-parsel-python","type":"article","title":"Guide to Parsel - the Best HTML Parsing in Python","description":"Learn to extract data from websites with Python and Parsel which is a Python library for HTML parsing through CSS selectors and XPath.","tags":["data-parsing","parsel"]},{"path":"/blog/posts/guide-to-tiktok-api","type":"article","title":"A Comprehensive Guide to TikTok API","description":"Discover TikTok's powerful APIs for developers, businesses, and researchers. Learn about their features, use cases, and access requirements.","tags":["api"]},{"path":"/blog/posts/what-is-http-401-error-and-how-to-fix-it","type":"article","title":"What is HTTP 401 Error and How to Fix it","description":"Learn about HTTP 401 status code meaning, causes, and solutions in this comprehensive guide. How to handle 401 unauthorized errors effectively","tags":["http"]},{"path":"/blog/posts/jsonl-vs-json","type":"article","title":"JSONL vs JSON","description":"Learn the differences between JSON and JSONLines, their use cases, and efficiency. Why JSONLines excels in web scraping and real-time processing","tags":["data-parsing"]},{"path":"/blog/posts/guide-to-local-llm","type":"article","title":"How to Run a Local LLM with Ollama","description":"Run a local LLM with Ollama in 2026, choose a model that fits your memory, and connect web data through LangChain with a clear cloud boundary.","tags":["ai"]},{"path":"/blog/posts/guide-to-seleniumbase-better-selenium","type":"article","title":"Guide to SeleniumBase — A Better & Easier Selenium","description":"Learn about SeleniumBase browser automation with simple syntax, cross-browser support, and robust features, perfect for testing and web scraping.","tags":["headless-browser","selenium"]},{"path":"/blog/posts/web-scraping-java-jsoup-html-parsing","type":"article","title":"Web Scraping and HTML Parsing with Jsoup and Java","description":"Master web scraping with jsoup — a Java library for scraping and parsing HTML. Learn how to extract and manipulate data and handle limitations.","tags":["data-parsing","java"]},{"path":"/blog/posts/php-84-new-dom-selector","type":"article","title":"Guide to PHP 8.4 new DOM Selector Feature","description":"Learn about PHP 8.4’s new DOM Selector feature. Simplify DOM manipulation using intuitive CSS selectors for cleaner, more efficient code.","tags":["php"]},{"path":"/blog/posts/guide-to-curl-ignore-ssl-errors","type":"article","title":"How to Ignore cURL SSL Errors","description":"Learn to handle SSL errors in cURL, including using self-signed certificates, ignore options. Explore common issues, safe cURL practices.","tags":["curl","http"]},{"path":"/blog/posts/guide-to-okhttp-java-kotlin","type":"article","title":"Comprehensive Guide to OkHttp for Java and Kotlin","description":"Discover OkHttp, a powerful HTTP client for Java and Kotlin. Explore its features, setup, advanced usage, and solutions to common errors.","tags":["http","tools"]},{"path":"/blog/posts/everything-to-know-about-web-scraping-python","type":"article","title":"Everything to Know to Start Web Scraping in Python Today","description":"Ultimate modern intro to web scraping using Python. How to scrape data using HTTP or headless browsers, parse it using AI and scale and deploy.","tags":["python"]},{"path":"/blog/posts/guide-to-error-code-520-cloudflare-and-fixes","type":"article","title":"Guide to Cloudflare's Error Code 520 and How to Fix it","description":"Learn about Cloudflare's infamous error code 520. How can it be addressed from your client or server point of view and what does it mean.","tags":["http"]},{"path":"/blog/posts/guide-to-ssl-error-meaning-and-fixes","type":"article","title":"Guide to SSL Errors: What do they mean and how to fix them","description":"Guide to all SSL error issues. How to resolve SSL connection error in your server or cover of each SSL error meaning and resolution guide","tags":["http"]},{"path":"/blog/posts/what-is-cloudflare-1015-error-and-how-to-fix-it","type":"article","title":"What is Error 1015 (Cloudflare) and How to Fix it?","description":"An in-depth exploration of error 1015 to understand why you're being rate-limited by Cloudflare and discover effective strategies to resolve it.","tags":["blocking","http"]},{"path":"/blog/posts/guide-to-google-finance-api","type":"article","title":"Google Finance API and the Best Alternatives for Developers (2026)","description":"Learn about Google Finance data and it's the officially discontinued Google Finance API and Google Finance data alternatives and secret access.","tags":["api"]},{"path":"/blog/posts/guide-to-linkedin-api-and-alternatives","type":"article","title":"LinkedIn API Access: What You Can Build in 2026","description":"See which LinkedIn APIs are self-service, which need approval, what each product can access, and when public-page extraction is the better fit.","tags":["api"]},{"path":"/blog/posts/what-is-http-412-error-precondition-failed","type":"article","title":"What HTTP Error 412 Precondition Failed and How to Fix it?","description":"Discover the meaning of the HTTP 412 Precondition Failed error, its common causes in development and scraping, and learn how to troubleshoot it.","tags":["api","http"]},{"path":"/blog/posts/json-vs-xml","type":"article","title":"JSON vs XML: Key Differences and Modern Uses","description":"JSON and XML are two major data formats encountered in web development — here's how they differ and which is one better for your use case.","tags":["data-parsing"]},{"path":"/blog/posts/guide-to-yelp-api","type":"article","title":"How to Use Yelp API to Extract Business and Review Data","description":"Take an extensive look into Yelp API, its key features, pricing, and limitations. How to start with Yelp API and alternatives for Yelp data.","tags":["api"]},{"path":"/blog/posts/what-is-http-error-503-service-unavailable","type":"article","title":"HTTP Error 503 Service Unavailable and How to Fix it?","description":"Learn about HTTP 503 errors, their common causes, potential signs of blocking, and practical ways to handle them effectively.","tags":["api","blocking","http"]},{"path":"/blog/posts/guide-to-walmart-api","type":"article","title":"In-Depth Guide to the Walmart API","description":"Discover Walmart's robust API ecosystem, designed to streamline operations for sellers, suppliers, and partners and how to access these tools.","tags":["api","ecommerce"]},{"path":"/blog/posts/what-is-charles-proxy-and-how-to-use-it","type":"article","title":"What is Charles Proxy and How to Use it?","description":"Learn about of the most popular web debugging proxies — Charles Proxy and how to use it to intercept and analyze web traffic and web data.","tags":["proxies","tools"]},{"path":"/blog/posts/how-to-python-requests-post","type":"article","title":"Guide to Python requests POST method","description":"Discover how to use Python's requests library for POST requests, including JSON, form data, and file uploads, along with response handling tips.","tags":["http","python","requests"]},{"path":"/blog/posts/what-is-http-error-429-too-many-requests","type":"article","title":"What is HTTP Error 429 Too Many Request and How to Fix it","description":"Learn about HTTP status code 429 which is all about request throttling or distribution. See how to fix it with proxy and fingerprint rotation.","tags":["blocking","http"]},{"path":"/blog/posts/axios-vs-fetch","type":"article","title":"Axios vs Fetch: Which HTTP Client to Choose in JS?","description":"Discover the key differences between Fetch and Axios for making HTTP requests in JavaScript, determining which \nbest suits your project’s needs.","tags":["api","http","nodejs"]},{"path":"/blog/posts/python-requests-headers-guide","type":"article","title":"How to Set and Manage Headers in Python Requests in 2026","description":"set, inspect, and troubleshoot HTTP headers in Python Requests, including Session defaults, PreparedRequest, Authorization, and Content-Type handling.","tags":["http","python","requests"]},{"path":"/blog/posts/403-forbidden-web-scraping","type":"article","title":"How to Fix 403 Forbidden Errors When Web Scraping","description":"Getting 403 forbidden when web scraping? Learn 7 Python fixes from headers and user-agents to proxy rotation and TLS fingerprinting. Code examples included.","tags":["blocking","http","python"]},{"path":"/blog/posts/what-is-http-415-error-unsupported-media-type","type":"article","title":"What is HTTP 415 Error? (Unsupported Media Type)","description":"415 Unsupported Media Type error is usually caused by a misconfigured Content-Type header. Learn how to fix 415 http code with correct requests","tags":["http"]},{"path":"/blog/posts/what-is-http-422-error-unprocessable-entity","type":"article","title":"What is HTTP 422 Error? (Unprocessable Entity)","description":"422 Unprocessable Entity error is usually caused by a semantically invalid request. Learn http error 422 causes and how to fix your requests.","tags":["http"]},{"path":"/blog/posts/what-is-http-409-status-code-conflict","type":"article","title":"What is HTTP 409 Error? (Conflict)","description":"409 Conflict error is usually caused by conflicts between request data and the current state of a resource. Learn how to prevent 409 http errors.","tags":["http"]},{"path":"/blog/posts/http-error-413-payload-too-large","type":"article","title":"What is HTTP 413 Error? (Payload Too Large)","description":"Learn about HTTP 413 - error status code that indicates that POST or PUT request contains too much data. See how to fix it or bypass code 413.","tags":["http"]},{"path":"/blog/posts/playwright-vs-selenium","type":"article","title":"Playwright vs Selenium","description":"Explore the key differences between Playwright and Selenium headless browser automation libraries. Which one is better for scraping and testing","tags":["headless-browser","playwright","selenium"]},{"path":"/blog/posts/what-is-http-error-406-not-acceptable","type":"article","title":"What is HTTP 406 Error? (Not Acceptable)","description":"406 Not Acceptable error is usually caused by a misconfigured Accept type header. Learn how to prevent 406 http code with correct request config","tags":["http"]},{"path":"/blog/posts/what-is-parsing-turning-data-into-insights","type":"article","title":"What is Parsing? From Raw Data to Insights","description":"Learn what parsing is, explore Python JSON, HTML, and PDF parsers, and discover how to parse data efficiently using modern tools and techniques.","tags":["ai","data-parsing","python"]},{"path":"/blog/posts/concurrency-vs-parallelism","type":"article","title":"Concurrency vs Parallelism","description":"Comparison of parallelism versus concurrency in programming. What's the difference and how and when to use each technique to scale up programs.","tags":["nodejs","python","scaling"]},{"path":"/blog/posts/how-to-use-curl-get-requests","type":"article","title":"How to Use cURL GET Requests","description":"Complete intro to how to use curl get requests with curl headers, curl authentication. How to use curl to retrieve any page and handle URL params","tags":["curl"]},{"path":"/blog/posts/browser-fingerprinting-with-creepjs","type":"article","title":"What is CreepJS Browser Fingerprint and How to Bypass It","description":"See how Creepjs generates browser fingerprint test and how to create antifingerprint browser of your own through browser fingerprint spoofing.","tags":["blocking"]},{"path":"/blog/posts/what-is-a-headless-browser","type":"article","title":"What is a Headless Browser? Top 5 Headless Browser Tools","description":"Overview of automation using real web browsers through top headless browser tools. Turn real browsers into web automation machines and scrapers.","tags":[]},{"path":"/blog/answers/how-to-take-screenshots-nodejs","type":"answer","title":"How to take screenshots in NodeJS?","description":"Learn how to screenshot in Node.js using Playwright & Puppeteer. Includes installation, concepts, and customization tips.","tags":["headless-browser","nodejs","playwright","puppeteer","screenshots"]},{"path":"/blog/posts/how-to-take-screenshots-in-python","type":"article","title":"How To Take Screenshots In Python?","description":"Learn how to take Python screenshots through Selenium and Playwright, including common browser tips and tricks for customizing web page captures.","tags":["headless-browser","python","screenshots"]},{"path":"/blog/posts/what-is-the-best-screenshot-api","type":"article","title":"6 Best Screenshot APIs in 2026","description":"Compare 6 screenshot APIs by use case, entry price and anti-bot support. See which capture protected pages and how often each failed in our test.","tags":["screenshots"]},{"path":"/blog/posts/web-scraping-with-go","type":"article","title":"Web Scraping with Go: Colly, goquery, and Browser Tools in 2026","description":"Web scraping with Go in 2026: compare libraries for static pages, crawls, and JavaScript, with tested net/http, goquery, Colly, and chromedp code.","tags":["data-parsing","frameworks","golang","http"]},{"path":"/blog/posts/how-to-use-web-scaping-for-rag-applications","type":"article","title":"How to Power-Up LLMs with Web Scraping and RAG","description":"How to use LLM and web scraping for RAG applications using either LlamaIndex or LangChain. In depth step-by-step Python tutorial.","tags":["ai","project","python"]},{"path":"/blog/posts/web-scraping-with-cloud-browsers","type":"article","title":"Web Scraping With Cloud Browsers","description":"Intro to cloud browsers, their benefits, a step-by-step setup with self-hosted Selenium-grid cloud browsers. How to bypass cloud browser blocking","tags":["headless-browser","scaling"]},{"path":"/blog/posts/how-to-scrape-forms","type":"article","title":"How to Scrape Forms","description":"Introduction to web scraping forms through a step-by-step guide using HTTP clients and headless browsers like playwright, selenium and puppeteer.","tags":["headless-browser","httpx","nodejs","python","scrapeguide"]},{"path":"/blog/posts/how-to-build-minimum-advertised-price-monitoring-tool","type":"article","title":"How to Build Minimum Advertised Price (MAP) Monitoring Tool","description":"Tutorial on minimum advertised price monitoring through web scraping using Python and free tools for collecting price data and it's analysis.","tags":["project","python"]},{"path":"/blog/posts/how-to-scrape-reddit-social-data","type":"article","title":"How to Scrape Reddit Posts, Subreddits and Profiles","description":"Scrape Reddit posts, comments, subreddits, and profiles with Python. Use current repo-backed code, handle 403 blocks, and choose the right access method in 2026","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-with-headless-firefox","type":"article","title":"How to Scrape With Headless Firefox","description":"Tutorial on how to use headless Firefox with Selenium, Playwright, and Puppeteer for web scraping, including practical examples for each library.","tags":["headless-browser","nodejs","playwright","puppeteer","python","selenium"]},{"path":"/blog/answers/how-to-use-css-selectors-in-nim","type":"answer","title":"How to use CSS Selectors in Nim ?","description":"how to parse HTML using CSS selectors in Nim programming language using either CSS3Selectors or nimquery libraries.","tags":["css-selectors"]},{"path":"/blog/posts/how-to-use-tor-for-web-scraping","type":"article","title":"How to Use Tor as a Proxy for Web Scraping in 2026","description":"Learn about web scraping using Tor as a proxy and rotating proxy server by randomly changing the IP address with HTTP or SOCKS.","tags":[]},{"path":"/blog/posts/how-to-know-what-anti-bot-website-uses","type":"article","title":"How to Know What Anti-Bot Service a Website is Using?","description":"Learn how to use WhatWaf and Wafw00f to identify which WAF service is used on a website and how to avoid these services' detection when scraping.","tags":["blocking","tools"]},{"path":"/blog/posts/how-to-scrape-linkedin","type":"article","title":"How to Scrape LinkedIn Profiles, Companies, and Jobs in 2026","description":"LinkedIn blocks scrapers aggressively. Learn how to scrape profiles, companies, and jobs using ScrapFly's anti-bot solution with Python code.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-intercept-background-requests-with-selenium-wire","type":"article","title":"Selenium Wire Tutorial: Intercept Background Requests","description":"Learn web scraping with Selenium Wire. We'll define what it is, installing and using it to inspect and manipulate background requests.","tags":["headless-browser","python","selenium","tools"]},{"path":"/blog/posts/how-to-scrape-similarweb","type":"article","title":"How to Scrape SimilarWeb Website Traffic Analytics","description":"Learn how to scrape SimilarWeb. We'll scrape comprehensive domain traffic insights, websites comparing data, sitemaps, and trending domains.","tags":["python","scrapeguide","seo"]},{"path":"/blog/posts/how-to-scrape-bestbuy-product-offer-and-review-data","type":"article","title":"How to Scrape BestBuy Product, Offer and Review Data","description":"Learn how to scrape BestBuy for different data types from product, search, review, and sitemap pages using different web scraping techniques.","tags":["ecommerce","hidden-api","python","scrapeguide"]},{"path":"/blog/posts/sending-http-requests-with-curlie-a-better-curl","type":"article","title":"Sending HTTP Requests With Curlie: A better cURL","description":"Explore Curlie, a better version of cURL. You will learn how to use and configure it for sending HTTP requests through a step-by-step guide.","tags":["curl","http","tools"]},{"path":"/blog/answers/how-to-solve-the-curl-60-error-when-proxy","type":"answer","title":"How to Solve the cURL (60) Error When Using Proxy?","description":"The cURL (60) error is a common error encountered when using proxies with cURL. Learn what is the exact cause of this error and how to solve it.","tags":["curl","proxies"]},{"path":"/blog/answers/how-to-use-proxy-with-curl","type":"answer","title":"How To Use Proxy With cURL?","description":"Proxies are essential to avoid IP address blocking and accessing restricted web pages over a specific location. Learn how to proxies with cURL.","tags":["curl","proxies"]},{"path":"/blog/answers/what-is-the-curl-28-error","type":"answer","title":"What is The cURL (28) Error, Couldn't connect to server?","description":"The cURL (28) indicates a proxy connection error. This error arises when the cURL request can't connect to the proxy server.","tags":["curl","proxies"]},{"path":"/blog/answers/how-to-download-file-curl","type":"answer","title":"How To Download a File With cURL?","description":"cURL allows for downloading binary files using the cURL -O option here's how to use it effectively and common errors related to file downloads.","tags":["curl"]},{"path":"/blog/answers/how-to-follow-redirects-in-curl","type":"answer","title":"How to Follow Redirects In cURL?","description":"Redirects are caused by HTTP pages moving to a different location. They can be handled automatically or explicitly - here's how to do it in cURL.","tags":["curl"]},{"path":"/blog/answers/how-to-send-a-post-request-using-curl","type":"answer","title":"How To Send cURL POST Requests?","description":"POST type requests send data to the web server which is popular http method for web interactions like search. Here's how to POST in cURL.","tags":["curl"]},{"path":"/blog/answers/how-to-send-curl-head-requests","type":"answer","title":"How to Send a HEAD Request With cURL?","description":"The HEAD HTTP method is used to gather information and metadata about a specific resource. Learn how to send HEAD requests with cURL.","tags":["curl"]},{"path":"/blog/answers/how-to-send-multiple-curl-requests-in-parallel","type":"answer","title":"How To Send Multiple cURL Requests in Parallel?","description":"To send request in parallel using cURL command line client the -Z or --parallel option can be used and mixed with other config options.","tags":["curl"]},{"path":"/blog/answers/how-to-set-authorization-with-curl-full-examples-guide","type":"answer","title":"How to Set cURL Authentication - Full Examples Guide","description":"Learn how to set basic authentication, bearer tokens, and cookie authentication with cURL through a step-by-step guide.","tags":["curl"]},{"path":"/blog/answers/how-to-set-curl-config-file","type":"answer","title":"How to Use cURL Config Files?","description":"cURL can be configured using config.txt files which can definite each cURL option. Then, the \"-K\" option can be used to provide your config.","tags":["curl"]},{"path":"/blog/answers/how-to-set-curl-user-agent","type":"answer","title":"How to Set User Agent With cURL?","description":"The User-Agent header is one of the essential headers which identifies the request sender's device. Learn how to set User-Agent with cURL.","tags":["curl"]},{"path":"/blog/posts/how-to-use-curl-for-web-scraping","type":"article","title":"How to Use cURL For Web Scraping","description":"Learn how to send HTTP requests with cURL and how to use cURL for web scraping, such as scraping dynamic pages and avoiding getting blocked.","tags":["curl","http","tools"]},{"path":"/blog/posts/how-to-scrape-tiktok-python-json","type":"article","title":"How To Scrape TikTok in 2026","description":"Scrape TikTok in 2026 with new anti-bot defenses and hidden APIs. Extract profiles, videos, comments, and search data reliably.","tags":["hidden-api","python","scrapeguide","tools"]},{"path":"/blog/answers/how-to-copy-as-curl-with-brave","type":"answer","title":"How to Copy as cURL With Brave?","description":"Brave allows for capturing HTTP requests on web pages. Learn how to use brave's developer tools to copy the requests as cURL.","tags":["curl","http"]},{"path":"/blog/answers/how-to-copy-as-curl-with-chrome","type":"answer","title":"How To Copy as cURL With Google Chrome?","description":"Google Chrome allows for capturing HTTP requests on web pages. Learn how to use Chrome's developer tools to the requests as cURL.","tags":["curl","http"]},{"path":"/blog/answers/how-to-copy-as-curl-with-edge","type":"answer","title":"How to Copy as cURL With Edge?","description":"Edge allows for capturing HTTP requests on web pages. Learn how to use Edge's developer tools to copy requests as cURL.","tags":["curl","http"]},{"path":"/blog/answers/how-to-copy-as-curl-with-firefox","type":"answer","title":"How to Copy as cURL With Firefox?","description":"Firefox allows for capturing HTTP requests on web pages. Learn how to use Firefox's developer tools to copy the requests as cURL.","tags":["curl","http"]},{"path":"/blog/answers/how-to-copy-as-curl-with-safari","type":"answer","title":"How to Copy as cURL With Safari?","description":"Safari allows for capturing HTTP requests on web pages. Learn how to use Safari's developer tools to copy requests as cURL.","tags":["curl","http"]},{"path":"/blog/posts/how-to-use-scrapy-with-playwright","type":"article","title":"Scrapy Playwright Tutorial: Scrape Dynamic Websites","description":"Learn how to use scrapy-playwright with Scrapy 2.18 to scrape JavaScript pages, wait for dynamic content, manage browser contexts, and run on Windows.","tags":["headless-browser","playwright","python","scrapy"]},{"path":"/blog/posts/web-scraping-dynamic-web-pages-with-scrapy-selenium","type":"article","title":"Scrapy Selenium Guide: JavaScript Pages With Selenium 4","description":"Use Scrapy 2.18 with Selenium 4 via scrapy-selenium4. Pinned setup, explicit waits, bounded scrolling, form login, screenshots, and driver cleanup.","tags":["headless-browser","python","scrapy","selenium"]},{"path":"/blog/posts/web-scraping-with-scrapy-splash","type":"article","title":"Scrapy Splash Guide: Setup, Lua, and Compatibility","description":"Maintain Scrapy Splash with a pinned stack, current request fingerprinting, working Lua form submission, and clear Scrapy 2.18 compatibility limits.","tags":["frameworks","headless-browser","python","scrapy"]},{"path":"/blog/posts/how-to-track-competitor-pricing-using-web-scraping","type":"article","title":"How to Track Competitor Prices Using Web Scraping","description":"Learn how to create a tool for tracking competitor prices using Python by scraping products from different providers and comparing their prices.","tags":["ecommerce","project","python"]},{"path":"/blog/posts/intro-to-using-web-scraping-for-sentiment-analysis","type":"article","title":"Intro to Using Web Scraping For Sentiment Analysis","description":"Learn how to perform sentiment analysis with web scraping using TextBlob and Huggingface on web-scraped data through a step-by-step guide.","tags":["ai","project","python"]},{"path":"/blog/posts/using-api-clients-for-web-scraping-postman","type":"article","title":"Using API Clients For Web Scraping: Postman","description":"Learn to use API clients for web scraping. Locate hidden requests, manipulate them with Postman, and build efficient API-based web scrapers.","tags":["hidden-api","tools"]},{"path":"/blog/posts/intro-to-parsing-html-xml-python-lxml","type":"article","title":"Python lxml Tutorial: How to Parse HTML and XML","description":"Learn what lxml is, how to install it and using lxml for parsing HTML and XML files. lxml example in web scraping and extracting product data","tags":["data-parsing","python","tools"]},{"path":"/blog/posts/curl-impersonate-scrape-chrome-firefox-tls-http2-fingerprint","type":"article","title":"Use Curl Impersonate to scrape as Chrome or Firefox","description":"Learn what Curl Impersonate is, how it works, how to install and use it. Finally, we'll explore using it with Python to avoid scraping blocking.","tags":["blocking","curl","http","tools"]},{"path":"/blog/posts/flaresolverr-guide","type":"article","title":"FlareSolverr Guide for Web Scraping: Setup, API, Proxies, and Troubleshooting","description":"Learn how to install and use FlareSolverr for web scraping, including API requests, sessions, proxies, common errors, limitations, and alternatives.","tags":["blocking","http","python","tools"]},{"path":"/blog/posts/how-to-use-cloudproxy-for-web-scraping","type":"article","title":"CloudProxy Web Scraping: A 2026 Reality Check","description":"Set up CloudProxy with current Python code, calculate the real VM cost, and decide when cloud datacenter proxies are still the right fit for scraping.","tags":["proxies","tools"]},{"path":"/blog/posts/how-to-use-browser-extensions-with-playwright-puppeteer-and-selenium","type":"article","title":"How to use Headless Chrome Extensions for Web Scraping","description":"Learn how to use browser extensions with headless browser libraries. You will also learn about useful Chrome extensions for web scraping.","tags":["nodejs","playwright","puppeteer","python","selenium","tools"]},{"path":"/blog/posts/how-to-use-cache-in-web-scraping","type":"article","title":"How to Use Cache In Web Scraping for Major Performance Boost","description":"Intro to web scraping caches. How caching can significantly reduce scraping costs and drastically improve performance and redis cache example.","tags":["python","tools"]},{"path":"/blog/posts/how-to-parse-xml","type":"article","title":"How to Parse XML","description":"How to parse XML using CSS selectors, XPath and language native tools in Python, Php, javascript and other languages. Complete XML parsing guide.","tags":["css-selectors","data-parsing","python","xpath"]},{"path":"/blog/posts/how-to-build-a-price-tracker-in-python","type":"article","title":"How to Build a Price Tracker Using Python","description":"Build a Python price tracker that scrapes product prices, stores history in SQLite, detects price drops with a configurable threshold, and runs on a schedule with APScheduler or cron.","tags":["project","python"]},{"path":"/blog/posts/how-to-scrape-bing-search-using-python","type":"article","title":"How to Scrape Bing Search with Python","description":"Learn how to scrape Bing using Python. You will also learn how to overcome its scraping challenges, such as the complex HTML structure and blocking.","tags":["python","scrapeguide","seo"]},{"path":"/blog/posts/how-to-bypass-captcha-web-scraping","type":"article","title":"5 Proven Ways to Bypass CAPTCHA in Python for 2026","description":"Stop CAPTCHAs before they block your scraper. Learn 5 Python methods for avoiding and solving CAPTCHA challenges in web scraping.","tags":["blocking"]},{"path":"/blog/posts/how-to-bypass-kasada-anti-scraping-waf","type":"article","title":"How to Bypass Kasada Anti-Bot When Web Scraping in 2026","description":"Learn what Kasada is and how it's used to block bots such as web scrapers. You will also learn how to bypass Kasada blocking while scraping.","tags":["blocking"]},{"path":"/blog/posts/how-to-scrape-g2-company-data-and-reviews","type":"article","title":"How to Scrape G2 Company Data and Reviews","description":"Tutorial on how to scrape G2.com using Python. Scrape reviews, company data, search pages, product data and alternatives without being blocked.","tags":["python","scrapeguide"]},{"path":"/blog/posts/what-are-honeypots-and-how-to-avoid-them","type":"article","title":"What are Honeypots and How to Avoid Them in Web Scraping","description":"Introduction to web honeypots, their types and functions and how they are used to identify and block web scrapers and bots and how to avoid them.","tags":["blocking"]},{"path":"/blog/posts/how-to-scrape-etsy-com-product-review-data","type":"article","title":"How to Scrape Etsy.com Product, Shop and Search Data","description":"tutorial on how to scrape Etsy data from product, shop and search pages using Python. How to bypass Etsy blocking and parse HTML and JSON data.","tags":["ecommerce","project","python","scrapeguide"]},{"path":"/blog/posts/how-to-hide-your-ip-address-while-scraping","type":"article","title":"How to Hide Your IP Address","description":"Learn about bout IP addresses, what they are and why hide them. We'll also explore four essential alternatives you can use to hide your IP address.","tags":["blocking","proxies","tools"]},{"path":"/blog/posts/how-to-scrape-trustpilot-com-reviews","type":"article","title":"How to Scrape Trustpilot.com Reviews and Company Data","description":"Learn how to scrape trustpilot.com company details and reviews without getting blocked. You will also learn to use the trustpilot.com private API.","tags":["hidden-api","python","scrapeguide"]},{"path":"/blog/posts/web-scraping-to-google-sheets","type":"article","title":"Web Scraping to Google Sheets","description":"Learn about web scraping to Google Sheets, how to access and store data on Google Sheets using Python, in a real-life project example.","tags":["data-parsing","project","python"]},{"path":"/blog/posts/how-to-scrape-domain-com-au-real-estate-property-data","type":"article","title":"How to Scrape Domain.com.au Real Estate Property Data","description":"Tutorial for creating a python scraper for domain.com.au real estate property data collector. How to find real estate properties and scrape it.","tags":["python","real-estate","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-realestate-com-au-property-listing-data","type":"article","title":"How to Scrape Realestate.com.au Property Listing Data","description":"Learn how to scrape realestate.com.au for real estate data from property and search pages and how to avoid realestate.com.au scraping blocking.","tags":["python","real-estate","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-immowelt-de-real-estate-properties","type":"article","title":"How to Scrape Immowelt.de Real Estate Data","description":"Tutorial for scraping immowelt.de. Step by step guide for creating Immowelt web scraper without the need of an Immowelt API for free.","tags":["python","real-estate","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-homegate-ch-real-estate-property-data","type":"article","title":"How to Scrape Homegate.ch Real Estate Property Data in 2026","description":"Scrape Homegate.ch property listings and search results with Python. Updated 2026 tutorial covering JavaScript rendering, pagination, and anti-bot bypass.","tags":["python","real-estate","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-immobilienscout24","type":"article","title":"How to Scrape ImmobilienScout24.de Real Estate Data (2026)","description":"how to scrape immobilienscout24.de without blocking for real estate data in germany. Web scraping guide for Python through HTTP and HTML parsing.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-immoscout24-ch-real-estate-property-data","type":"article","title":"How to Scrape Immoscout24.ch Real Estate Property Data","description":"Tutorial for web scraping immoscout24 using Python and private API and hidden JSON data techniques. How to bypass blocking and scrape real estate","tags":["python","real-estate","scrapeguide"]},{"path":"/blog/posts/how-to-handle-cookies-in-web-scraping","type":"article","title":"How to Handle Cookies  in Web Scraping","description":"Tutorial for cookies in web scraping. What are they and how to take advantage of cookie process to authenticate or set website preferences.","tags":["http"]},{"path":"/blog/posts/how-to-scrape-seloger-com-listing-real-estate-ads","type":"article","title":"How to Scrape Seloger.com - Real Estate Listing Data","description":"Tutorial on seloger.com web scraping and how to avoid scraper blocking. Use Python to scrape seloger.com real estate property listings and search","tags":["project","python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-leboncoin-marketplace-real-estate","type":"article","title":"How to Scrape Leboncoin.fr in 2026 (No API Needed)","description":"Learn how to scrape Leboncoin.fr in 2026 with Python. Extract listing data from search results and individual ad pages using hidden JSON data, no API needed. Includes anti-bot bypass and pagination.","tags":["python","scrapeguide"]},{"path":"/blog/posts/intro-to-web-scraping-using-selenium-grid","type":"article","title":"Intro to Web Scraping Using Selenium Grid","description":"Intro to using Selenium Grid server for web scraping. How to set it up using docker and scrape with it concurrently. Real life example and faq.","tags":["python","selenium"]},{"path":"/blog/posts/how-to-scrape-hidden-apis","type":"article","title":"How to Scrape Hidden APIs","description":"Tutorial on web scraping hidden APIs. How dynamic websites load content through background request and how to see it and replicate it in Python.","tags":["api","python"]},{"path":"/blog/posts/web-scraping-without-blocking-using-undetected-chromedriver","type":"article","title":"Web Scraping Without Blocking With Undetected ChromeDriver","description":"Intro to Undetected ChromeDriver - selenium extension for bypassing many scraper blocking extensions. Hands-on tutorial and real-life example.","tags":["blocking","python","tools"]},{"path":"/blog/posts/how-to-scrape-emails-using-python","type":"article","title":"Web Scraping Emails using Python","description":"How to create a email scraping tool using Python. Intro to email address crawling and how to solve common challenges like email obfuscation.","tags":["data-parsing","project","python"]},{"path":"/blog/posts/how-to-scrape-phone-numbers-with-python","type":"article","title":"Web Scraping Phone Numbers with Python","description":"Deep dive into phone number crawling. We'll explore an example object and cover common phone number scraping challenges like obfuscation.","tags":["data-parsing","project","python"]},{"path":"/blog/posts/how-to-scrape-google-trends","type":"article","title":"How to Scrape Google Trends using Python","description":"Intro to web scraping Google Trends data using Python. What is Google Trends and what makes it a valuable web scraping target using Python.","tags":["python","seo"]},{"path":"/blog/posts/how-to-avoid-blocking-scraping-images","type":"article","title":"How to Avoid Scraper Blocking when Scraping Images","description":"Introduction to scraper blocking when it comes to image scraping. What are some popular scraper blocking techniques and how to avoid them.","tags":["blocking"]},{"path":"/blog/posts/how-to-web-scrape-images-from-websites-python","type":"article","title":"Intro to Web Scraping Images with Python","description":"Image web scraping tutorial with Python. How to scrape images using python and common challenges like hidden image data and dynamic js images.","tags":["crawling","data-parsing","python","scrapeguide"]},{"path":"/blog/posts/web-scraping-google-seo-keywords","type":"article","title":"How to Scrape Google SEO Keyword Data and Rankings","description":"Learn how to use SEO web scraping for SEO keyword optimization. You will also learn how to scrape Google search rankings and suggested keywords.","tags":["project","python","seo"]},{"path":"/blog/posts/xpath-cheatsheet","type":"article","title":"Ultimate XPath Cheatsheet for HTML Parsing in Web Scraping","description":"Complete cheatsheet for all XPath selector functions for HTML parsing in web scraping with real-life interactive examples and explanations.","tags":["data-parsing","xpath"]},{"path":"/blog/posts/user-agent-header-in-web-scraping","type":"article","title":"How to Effectively Use User Agents for Web Scraping","description":"A guide on using User-Agent headers for web scraping. How to set and rotate user agent headers in web scraping to avoid web scraping blocking.","tags":["http","python"]},{"path":"/blog/posts/observing-ecommerce-market-trends-with-web-scraping","type":"article","title":"How to Observe E-Commerce Trends using Web Scraping","description":"Web scraping project for scraping e-commerce data and observing market trends using visualization graphs and plots for free.","tags":["ecommerce","project","python"]},{"path":"/blog/posts/how-to-scrape-in-another-language-or-currency","type":"article","title":"How to Scrape in Another Language, Currency or Location","description":"In-depth look at scraping websites in specific languages and currencies using Playwright or HTTPX and Python through example projects.","tags":["headless-browser","http","python"]},{"path":"/blog/posts/refining-json-datasets-with-chatgpt-web-scraping","type":"article","title":"JSON Parsing Made Easy with ChatGPT in Web Scraping","description":"Intro to reshaping web scraped JSON datasets to cleaner, easier format using ChatGPT and JMESPath JSON parsing language adn Python.","tags":["ai","data-parsing","python"]},{"path":"/blog/posts/ultimate-intro-to-web-scraping-with-typescript","type":"article","title":"Complete Guide to Web Scraping using Typescript","description":"Complete guide for web scraping using Typescript and Javascript. How to use axios and cheerio to scrape any page and solve common scraping challenges","tags":["scrapeguide","typescript"]},{"path":"/blog/posts/finding-hidden-web-data-with-chatgpt","type":"article","title":"Finding Hidden Web Data with ChatGPT Web Scraping","description":"Learn how to scrape hidden web data using ChatGPT. Prompt LLMs for fextracting data hidden in page HTML, such as data embedded in JavaScript.","tags":[]},{"path":"/blog/answers/how-to-edit-local-storage-using-devtools","type":"answer","title":"How to edit Local Storage data using browser Devtools","description":"To edit Local Storage browser's developer tools, Application tab -> Storage -> Local Storage where each value is represented in key-value format.","tags":["tools"]},{"path":"/blog/answers/html-table-to-xlsx-python-beautifulsoup","type":"answer","title":"How to scrape HTML table to Excel Spreadsheet (.xlsx)?","description":"To scrape tables to Excel spreadsheet we can use bs4, requests and xlsxwriter packages for Python. Here's how.","tags":["data-parsing","python"]},{"path":"/blog/answers/httpx-vs-requests-vs-aiohttp","type":"answer","title":"Python httpx vs requests vs aiohttp - key differences","description":"When it comes to these 3 popular http client packages they have different strenghts. Here's how to choose the right fit.","tags":["http","httpx","python"]},{"path":"/blog/answers/mobile-vs-residential-proxies-whats-the-difference","type":"answer","title":"Mobile vs Residential Proxies - which to choose for scraping?","description":"For web scraping mobile or residential proxies are the best though fill different niches. Here's how to choose.","tags":["proxies"]},{"path":"/blog/answers/what-are-private-proxies-compared-to-shared","type":"answer","title":"What are private proxies and how are they used in scraping?","description":"Private proxies mean the proxy is owned by a single user (opposite to shared proxies) which can significantly improve scraping performance.","tags":["proxies"]},{"path":"/blog/answers/what-are-socks5-proxies-in-web-scraping","type":"answer","title":"What are SOCKS5 proxies and how they compare to HTTP proxies?","description":"SOCKS5 is the latest protocol version of SOCKS network routing protocol. Here's how it differs from HTTP.","tags":["proxies"]},{"path":"/blog/answers/what-are-some-phantomjs-alternatives","type":"answer","title":"What are some PhantomJS alternatives for automating browsers?","description":"PhantomJS is a popular web browser control and automation tool - here are 3 better modern alternatives.","tags":["http","tools"]},{"path":"/blog/answers/what-case-should-http-headers-be","type":"answer","title":"What case should HTTP headers be in? Lowercase or Pascal-Case?","description":"HTTP header names can be either in lowercase or Pascal-Case and it's important to choose the right case to prevent scraper blocking.","tags":["http"]},{"path":"/blog/answers/what-python-libraries-support-http2","type":"answer","title":"What Python libraries support HTTP2?","description":"HTTP2 is still relatively new protocol version that is not yet widely supported. Here are the options for HTTP2 client in Python.","tags":["http","python"]},{"path":"/blog/posts/finding-web-selectors-with-chatgpt","type":"article","title":"Find Web Elements with ChatGPT and XPath or CSS selectors","description":"Tutorial on parsing HTML with ChatGPT, finding XPath and CSS selectors using ChatGPT. We'll do web scraping with ChatGPT and BeautifulSoup.","tags":["ai","python"]},{"path":"/blog/posts/parsing-html-with-chatgpt-code-interpreter","type":"article","title":"Crafting Web Scrapers using ChatGPT Code Interpreter is Easy","description":"The new chatgpt code intrepreter feature is an ideal assistant for crafting web scrapers. Here's how it can be used to help with HTML parsing.","tags":["ai","python"]},{"path":"/blog/posts/what-is-local-storage-and-how-to-scrape-it","type":"article","title":"How to scrape Local Storage using Headless Browsers","description":"Introduction to scraping local storage - a key value store available in all browsers and used in many modern SPAs - all using headless browsers like playwright.","tags":["headless-browser"]},{"path":"/blog/answers/how-to-click-on-alert-dialog-in-playwright","type":"answer","title":"How to handle popup dialogs in Playwright?","description":"To handle alert-type pop ups in Playwright the on \"dialog\" event can be captured and interacted with in both Python and NodeJS playwright clients","tags":["playwright","python"]},{"path":"/blog/answers/how-to-click-on-alert-dialog-in-puppeteer","type":"answer","title":"How to handle popup dialogs in Puppeteer?","description":"To click on a popup dialog in Puppeteer the dialog even can be captured and interacted with using page.on(\"dialog\") method. Here's how to do it.","tags":["puppeteer"]},{"path":"/blog/answers/how-to-click-on-alert-dialog-in-selenium","type":"answer","title":"How to handle popup dialogs in Selenium?","description":"To click on a pop-up alert using Selenium the alert_is_present method can be used to wait for and interact with alerts. Here's how.","tags":["selenium"]},{"path":"/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-playwright","type":"answer","title":"How to click on cookie popups and modal alerts in Playwright?","description":"To click on modal popups like the infamous cookie conset alert we can either find and click the agree button or remove it entirely.  Here's how.","tags":["playwright"]},{"path":"/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-puppeteer","type":"answer","title":"How to click on cookie popups and modal alerts in Puppeteer?","description":"To handle modal popups like cookie consents in Puppeteer the popup can be closed through a button click or removed entirely. Here's how.","tags":["puppeteer"]},{"path":"/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-selenium","type":"answer","title":"How to click on cookie popups and modal alerts in Selenium?","description":"To click on modal alerts like cookie popups in Selenium we can either find the button and click it or remove the modal elements. Here's how.","tags":["selenium"]},{"path":"/blog/answers/how-to-edit-cookies-using-chrome-devtools","type":"answer","title":"How to edit cookies in Chrome devtools?","description":"To edit cookies in Chrome's devtools suite the application->cookies section can be used. Here's how.","tags":["tools"]},{"path":"/blog/posts/how-to-scrape-threads","type":"article","title":"How to scrape Threads by Meta using Python (2026 Update)","description":"Guide how to scrape Threads - new social media network by Meta and Instagram - using Python and popular libraries like Playwright and XHR capture","tags":["python","scrapeguide"]},{"path":"/blog/posts/web-scraping-background-requests-with-headless-browsers-and-python","type":"article","title":"Web Scraping Background Requests with Headless Browsers","description":"Intro to web scraping background requests of dynamic websites using a headless browser and request/response capture with Python and Playwright","tags":["headless-browser","python"]},{"path":"/blog/answers/how-to-block-resources-in-selenium","type":"answer","title":"How to block resources in Selenium and Python?","description":"To block http resources in selenium we need an external proxy. Here's how to setup mitmproxy to block requests and responses in Selenium.","tags":["selenium"]},{"path":"/blog/answers/how-to-capture-xhr-requests-selenium","type":"answer","title":"How to capture background requests and responses in Selenium?","description":"To capture background requests and response selenium needs to be extended with Selenium-wire. Here's how to do it.","tags":["selenium"]},{"path":"/blog/answers/how-to-install-mitmproxy-certificate","type":"answer","title":"How to install mitmproxy certificate on Chrome and Chromium?","description":"Here are 5 easy steps to install SSL certificates to enable HTTPS traffic capture in mitmproxy tool used for intercepting and analyzing HTTP.","tags":["http","tools"]},{"path":"/blog/answers/how-to-scroll-to-the-bottom-with-playwright","type":"answer","title":"How to scroll to the bottom of the page with Playwright?","description":"Learn how to scroll to the bottom of the page with Playwright using three distinct approaches for both Python and NodeJS clients.","tags":["playwright"]},{"path":"/blog/answers/how-to-scroll-to-the-bottom-with-puppeteer","type":"answer","title":"How to scroll to the bottom of the page with Puppeteer?","description":"To scrape to the very bottom of the page with Puppeteer the javascript evaluation feature can be used within a while loop. Here's how.","tags":["puppeteer"]},{"path":"/blog/answers/how-to-scroll-to-the-bottom-with-selenium","type":"answer","title":"How to scroll to the bottom of the page with Selenium?","description":"To scroll to the very bottom of the page the javascript evaluation feature can be used within a while loop. Here's how.","tags":["selenium"]},{"path":"/blog/posts/parsing-datetime-strings-with-python-and-dateparser","type":"article","title":"How to Parse Datetime Strings with Python and Dateparser","description":"Dateparser is a popular Python package for parsing datetime strings. Here's how it can be used in web scraping and how to avoid common problems.","tags":["data-parsing","python"]},{"path":"/blog/posts/top-10-web-scraping-libraries-in-python","type":"article","title":"7 Top Python Web Scraping Libraries in 2026","description":"Seven Python scraping libraries compared by the parsing, browser, crawling, fetching, JSON-query, and validation jobs they perform, plus an API for blocked pages","tags":["data-parsing","python","tools"]},{"path":"/blog/answers/how-to-use-proxies-nodejs-axios","type":"answer","title":"How to use proxies with NodeJS axios?","description":"To use proxies with axios and nodejs the proxy parameter of get and post methods can be used. Here's how.","tags":["http","nodejs"]},{"path":"/blog/answers/how-to-use-proxies-php-guzzle","type":"answer","title":"How to use proxies with PHP Guzzle?","description":"To use proxies with PHP Guzzle library the proxy parameter can be used which mirrors standard configuration patterns of cURL library.","tags":["http","php"]},{"path":"/blog/answers/how-to-use-proxies-python-httpx","type":"answer","title":"How to use proxies with Python httpx?","description":"To use proxies with Python's httpx library the proxies parameter can be used for http, https and socks5 proxies. Here's how.","tags":["http","httpx","python"]},{"path":"/blog/answers/how-to-scrape-images-from-website","type":"answer","title":"How to scrape images from a website?","description":"To scrape all images from a given website python with beautifulsoup and httpx can be used. Here's an example.","tags":["python"]},{"path":"/blog/answers/how-to-select-elements-by-attribute-value","type":"answer","title":"How to select elements by attribute value in XPath?","description":"To select HTML elements by attribute value the @ syntax can be used together with = or contains() functions. Here's how.","tags":["xpath"]},{"path":"/blog/answers/what-are-scrapy-middlewares-and-how-to-use-them","type":"answer","title":"What are scrapy middlewares and how to use them?","description":"Scrapy downloader middlewares can be used to intercept and update outgoing requests and incoming responses. Here's how to use them.","tags":["scrapy"]},{"path":"/blog/posts/web-scraping-with-python-httpx","type":"article","title":"How to Web Scrape with HTTPX and Python","description":"Intro to using Python's httpx library for web scraping. Proxy and user agent rotation and common web scraping challenges, tips and tricks.","tags":["httpx","python"]},{"path":"/blog/answers/how-to-use-puppeteer-stealth-what-does-it-do","type":"answer","title":"Getting started with Puppeteer Stealth","description":"Puppeteer-stealth is a popular plugin for Puppeteer browser automation library. It patches browsers to be less detectible. Here's how to get started.","tags":["puppeteer"]},{"path":"/blog/answers/what-are-scrapy-pipelines-and-how-to-use-them","type":"answer","title":"What are scrapy pipelines and how to use them?","description":"Scrapy pipelines can be used to extend scraped result data with new fields or validate the whole datasets. Here's how.","tags":["scrapy"]},{"path":"/blog/answers/how-to-add-headers-to-every-or-some-scrapy-requests","type":"answer","title":"How to add headers to every or some scrapy requests?","description":"To add headers to scrapy's request the `DEFAULT_REQUEST_HEADERS` settting or a custom request middleware can be used. Here's how.","tags":["http","scrapy"]},{"path":"/blog/answers/how-to-pass-parameters-to-scrapy-spiders-cli","type":"answer","title":"How to pass custom parameters to scrapy spiders?","description":"To pass custom parameters to scrapy spider there CLI argument -a can be used. Here's how and why is it such a useful feature.","tags":["scrapy"]},{"path":"/blog/answers/scrapy-spiders-proxy-rotation","type":"answer","title":"How to Rotate Proxies in Scrapy","description":"Rotate proxies in Scrapy with request metadata, authenticated gateways, or a raw proxy pool, including retries, sticky sessions, and exit-IP verification.","tags":["proxies","scrapy"]},{"path":"/blog/answers/how-to-use-headless-browsers-with-scrapy","type":"answer","title":"How to use headless browsers with scrapy?","description":"To use headless browser with scrapy a plugin like scrapy-playwright can be used. Here's how to use it and what are some other alternatives.","tags":["headless-browser","scrapy"]},{"path":"/blog/answers/how-to-pass-data-between-scrapy-callbacks","type":"answer","title":"How to pass data between scrapy callbacks in Scrapy?","description":"To pass data between scrapy callbacks when scraping multiple pages the Request.item can be used. Here's how.","tags":["scrapy"]},{"path":"/blog/answers/how-to-pass-data-from-start-request-to-callbacks-scrapy","type":"answer","title":"How to pass data from start_requests to parse callbacks in scrapy?","description":"To pass data between scrapy callbacks like start_request and parse the Request.meta attribute can be used. Here's how.","tags":["scrapy"]},{"path":"/blog/answers/how-to-select-elements-by-attribute-containing-value-css-selectors","type":"answer","title":"How to select elements by attribute using CSS selectors?","description":"To select elements by attribute the powerful attribute selector can be used which has several selection options. Here's how.","tags":["css-selectors"]},{"path":"/blog/answers/how-to-select-elements-by-class-css-selectors","type":"answer","title":"How to select elements by class using CSS selectors?","description":"To select elements by class the .class selector can be used. To select by exact class value the [class=\"exact value\"] can be used instead. Here's how.","tags":["css-selectors"]},{"path":"/blog/answers/how-to-select-elements-by-id-css-selectors","type":"answer","title":"How to select elements by ID using CSS selectors?","description":"To select elements that contain an ID the #id selector can be used. To select elements by exact ID the [id=\"some value\"] can be used. Here's how.","tags":["css-selectors"]},{"path":"/blog/answers/how-to-select-following-sibling-element-css-selectors","type":"answer","title":"How to select following siblings using CSS selectors?","description":"To select following sibling elements using CSS selectors the + and ~ operators can be used. Here's how.","tags":["css-selectors"]},{"path":"/blog/answers/how-to-select-preceding-sibling-element-css-selectors","type":"answer","title":"Is it possible to select preceding siblings using CSS selectors?","description":"It's not possible to select preceding sibling directly but there are easy alternatives that can be implemented to select preceding siblings.","tags":["css-selectors"]},{"path":"/blog/answers/what-are-scrapy-items-and-itemloaders","type":"answer","title":"What are scrapy Item and ItemLoader objects and how to use them?","description":"Scrapy's Item and ItemLoader classes are great way to structure dataset parsing logic. Here's how to use it.","tags":["scrapy"]},{"path":"/blog/answers/how-to-count-selectors-in-xpath-and-why","type":"answer","title":"How to count selections in XPath and why?","description":"To count number of selected elements by an XPath selector the count() function can be used. Here's how to do it and why it's useful.","tags":["xpath"]},{"path":"/blog/answers/how-to-get-name-of-selected-element-in-xpath","type":"answer","title":"How to get the name of an HTML element in XPath?","description":"To find the name of a selected HTML element with XPath the name() function can be used. Here's how and why is this useful.","tags":["xpath"]},{"path":"/blog/answers/how-to-join-values-in-xpath","type":"answer","title":"How to join values using XPath concat?","description":"To join values in XPath the concat() function can be used to concatenate strings into one string. Here's how.","tags":["xpath"]},{"path":"/blog/answers/how-to-reverse-expression-in-xpath","type":"answer","title":"How to reverse expressions in XPath?","description":"To reverse expressions and predicates in XPath the not() function can be used. Here's how and why it's so useful.","tags":["xpath"]},{"path":"/blog/answers/how-to-select-elements-by-attribute-value-in-xpath","type":"answer","title":"How to select element with one of many names in XPath?","description":"To select an element with name matching one from an array of names the name() method can be used. Here's how.","tags":["xpath"]},{"path":"/blog/answers/how-to-select-elements-by-id-in-xpath","type":"answer","title":"How to select elements by ID in XPath?","description":"To select elements by ID attribute in XPath we can directly match it using = operator in a predicate or contains() function. Here's how.","tags":["xpath"]},{"path":"/blog/answers/how-to-select-elements-of-any-name-using-wildcards-in-xpath","type":"answer","title":"How to select any element using wildcard in XPath?","description":"To select any element the wildcard \"*\" axis selector can be used which will select any HTML element of any name within the current context.","tags":["xpath"]},{"path":"/blog/answers/how-to-select-elements-of-specific-position-in-xpath","type":"answer","title":"How to select elements of a specific position in XPath?","description":"To select elements of a specific position the position() function can be used in a selection predicate. Here's how.","tags":["xpath"]},{"path":"/blog/answers/how-to-select-last-element-in-xpath","type":"answer","title":"How to select last element in XPath?","description":"To select last element in XPath we cannot use indexing as -1 index is not supported. Instead, last() function can be used. Here's how.","tags":["data-parsing","xpath"]},{"path":"/blog/answers/how-to-select-sibling-elements-using-xpath","type":"answer","title":"How to select sibling elements in XPath?","description":"To select sibling elements in XPath the preceding-sibling and following-sibling axis can be used. Here's how and why it's so useful.","tags":["xpath"]},{"path":"/blog/answers/how-to-check-for-element-in-playwright","type":"answer","title":"How to check if element exists in Playwright?","description":"To check whether an HTML element is present on the page using Playwright the page.locator() method can be used. Here's how.","tags":["playwright","python"]},{"path":"/blog/answers/how-to-select-all-elements-between-two-known-elements-in-xpath","type":"answer","title":"How to select all elements between two elements in XPath?","description":"To select all elements between two different elements preceding-sibling or following-sibling axis selectors can be used. Here's how.","tags":["data-parsing","xpath"]},{"path":"/blog/answers/how-to-select-dictionary-key-recursively-in-python","type":"answer","title":"How to select dictionary key recursively in Python?","description":"To select dictionary keys recursively in Python the \"nested-lookup\" package implements the most popular nested key selection algorithms.","tags":["data-parsing","python"]},{"path":"/blog/answers/what-are-some-ways-to-parse-json-datasets-in-python","type":"answer","title":"What are some ways to parse JSON datasets in Python?","description":"There are several popular options when it comes to JSON dataset parsing in Python. The most popular packages are Jmespath and Jsonpath.","tags":["data-parsing","python"]},{"path":"/blog/posts/stepping-into-footwear-market-with-web-scraping","type":"article","title":"Stepping into Footwear Market with Web Scraping","description":"Introduction to data analytics with web scraped data. Tracking luxury footwear market using web scraping, python and basic data analytics.","tags":["fashion"]},{"path":"/blog/answers/what-is-asynchronous-web-scraping","type":"answer","title":"What is Asynchronous Web Scraping?","description":"Asynchronous programming is an accessible way to scale around IO blocking  which is especially powerful in web scraping. Here's why.","tags":["http"]},{"path":"/blog/posts/how-to-scrape-goat-com-fashion-apparel","type":"article","title":"How to Scrape Goat.com for Fashion Apparel Data in Python","description":"How to scrape Goat.com for new and second-hand apparel product data using Python and how to avoid blocking when scaling up goat.com crawlers.","tags":["ecommerce","fashion","python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-fashionphile","type":"article","title":"How to Scrape Fashionphile for Second Hand Fashion Data","description":"Easy step by step tutorial for scraping Fashionphile.com product data with Python and ScrapFly without getting blocked.","tags":["ecommerce","fashion","python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-sitemaps","type":"article","title":"How to Scrape Sitemaps to Discover Scraping Targets","description":"Scraping sitemaps can be an easy way to discover scrape targets and scrape all pages of the website. This tutorial covers how to scrape them.","tags":["crawling","data-parsing","nodejs","python"]},{"path":"/blog/posts/how-to-scrape-vestiairecollective","type":"article","title":"How to Scrape Vestiaire Collective for Fashion Product Data","description":"Guide on how to scrape the biggest 2nd hand luxury fashion marketplace Vestiaire Collective using Python and hidden web data scraping approach.","tags":["ecommerce","fashion","python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-nordstrom","type":"article","title":"How to Scrape Nordstrom Fashion Product Data","description":"Tutorial for web scraping Nordstrom.com using Python, httpx and parsel. Extracting hidden product data and how to parse it using jmespath.","tags":["ecommerce","fashion","python","scrapeguide"]},{"path":"/blog/answers/browser-developer-tools-in-web-scraping","type":"answer","title":"What are devtools and how they're used in web scraping?","description":"Developer tools suite is used in web development but can also be used in web scraping to understand how target websites work. Here's how to use it.","tags":["css-selectors","data-parsing","hidden-api","http","xpath"]},{"path":"/blog/posts/how-to-scrape-stockx","type":"article","title":"How to Scrape StockX e-commerce Data with Python","description":"Intro to scraping stockX.com product data using Python for free. How to collect stockx data using web scraping and avoid being blocked.","tags":["ecommerce","fashion","python","scrapeguide"]},{"path":"/blog/answers/how-to-use-curl-in-python","type":"answer","title":"How to use cURL in Python?","description":"cURL through libcurl is a popular library used in HTTP connections and can be used with Python through wrapper libraries like pycurl.","tags":["http","python"]},{"path":"/blog/answers/http-cookies-in-web-scraping","type":"answer","title":"What is HTTP cookies role in web scraping?","description":"HTTP cookies play a big role in web scraping. They can be used to configure website preferences and  play an important role in scraper detection.","tags":["http"]},{"path":"/blog/answers/http-vs-https-in-web-scraping","type":"answer","title":"HTTP vs HTTPS in web scraping ?","description":"HTTPS is a secure version of the HTTP protocol which can complicate the web scraping process in many different ways. Here's what it means.","tags":["http"]},{"path":"/blog/answers/ipv4-vs-ipv6-in-web-scraping","type":"answer","title":"What is the difference between IPv4 vs IPv6 in web scraping?","description":"IPv4 and IPv6 are two competing Internet Protocol version that have different advantages when it comes to web scraping. Here's what they are.","tags":["http","proxies"]},{"path":"/blog/answers/vpn-as-proxies-in-web-scraping","type":"answer","title":"How to use VPNs as proxies for web scraping","description":"VPNs can be used as IP proxies in web scraping. Here's how and what to keep an eye on when using this approach.","tags":["proxies"]},{"path":"/blog/answers/what-is-curl-and-how-is-it-used-in-web-scraping","type":"answer","title":"What is cURL and how is it used in web scraping?","description":"cURL is the most popular HTTP client and library (libcurl) that implements most of HTTP features meaning it's a powerful web scraping tool too.","tags":["http"]},{"path":"/blog/answers/what-is-mitm-proxy-and-how-is-it-used-web-scraping","type":"answer","title":"What is MITM and how is it used in web scraping?","description":"MITM tools can be used to intercept and modify http traffic of various applications like web browser or phone apps in web scraper development.","tags":["http"]},{"path":"/blog/posts/how-to-bypass-imperva-incapsula-anti-scraping","type":"article","title":"How to Bypass Imperva Incapsula when Web Scraping in 2026","description":"Depp look at how to add Imperva Incampusa bypass to web scrapers. How does it detect web scraping and how to avoid avoiding being blocked.","tags":["blocking"]},{"path":"/blog/posts/how-to-bypass-datadome-anti-scraping","type":"article","title":"How to Bypass Datadome Anti Scraping in 2026","description":"Indepth look at how to add Datadome bypass to web scrapers. How does it detect web scraping and how to avoid avoiding being blocked.","tags":["blocking"]},{"path":"/blog/posts/how-to-bypass-akamai-anti-scraping","type":"article","title":"How to Bypass Akamai when Web Scraping in 2026","description":"Deep look at how to add Akamai Bot Manager bypass to web scrapers. How does it detect web scrapers and best practices for avoiding being blocked.","tags":[]},{"path":"/blog/posts/how-to-bypass-perimeterx-human-anti-scraping","type":"article","title":"How to Bypass PerimeterX when Web Scraping in 2026","description":"In-depth look at how PerimeterX is detecting web scrapers and bots. How to bypass PerimeterX when web scraping and avoid being blocked.","tags":["blocking"]},{"path":"/blog/posts/how-to-bypass-cloudflare-anti-scraping","type":"article","title":"How to Bypass Cloudflare When Web Scraping in 2026","description":"Overview of how Cloudflare bot management service blocks web scrapers and how to bypass Cloudflare when web scraping by managing fingerprinting.","tags":["blocking"]},{"path":"/blog/posts/web-scraping-microformats","type":"article","title":"Web Scraping Simplified - Scraping Microformats","description":"Introduction to web scraping microformats - special data markup powered by schema.org. How to scrape microformats in Python and an example project.","tags":["data-parsing","python"]},{"path":"/blog/posts/how-to-scrape-twitter","type":"article","title":"How to Scrape Twitter (X.com) with Python in 2026","description":"Scrape Twitter (X.com) with Python in 2026: how guest tokens and doc_ids work, why snscrape and Twint died, and working code for profiles and tweets.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-rightmove","type":"article","title":"How to Scrape Rightmove Real Estate Listings (2026 Guide)","description":"Learn to scrape Rightmove property data with Python. Extract prices, addresses, photos, and agent details from a single HTTP request per page.","tags":["api","python","real-estate","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-google","type":"article","title":"How to Scrape Google Search Results in 2026","description":"Tutorial on scraping Google search using Python with a few community packages. How to parse dynamic SERP HTML and how to not get blocked.","tags":["python","scrapeguide"]},{"path":"/blog/posts/parse-json-jsonpath-python","type":"article","title":"Introduction to Parsing JSON with  Python JSONPath","description":"Intro to using Python and JSONPath library and a query language for parsing JSON datasets. How to setup jsonpath and real life examples.","tags":["data-parsing","python"]},{"path":"/blog/posts/how-to-scrape-ebay","type":"article","title":"How to Scrape Ebay in 2026","description":"Learn how to scrape eBay product listings, multi-variant data and search results using Python and the Scrapfly SDK with anti-bot bypass.","tags":["python","scrapeguide"]},{"path":"/blog/answers/how-to-open-python-responses-in-browser","type":"answer","title":"How to open Python http responses in a web browser?","description":"To preview Python http responses we can use temporary files and the built-in webbrowser module. Here's how.","tags":["httpx","python","requests"]},{"path":"/blog/answers/playwright-in-ipython","type":"answer","title":"How to run Playwright in Jupyter notebooks?","description":"Learn why the synchronous execution of Playwright is blocked on Jupyter notebooks and how to solve it using asyncio.","tags":["jupyter","playwright"]},{"path":"/blog/answers/python-requests-exception-connecttimeout","type":"answer","title":"How to fix python requests ConnectTimeout error?","description":"Python's ConnectTimeout exception is caused when connection can't be established fast enough. Here's how to fix it.","tags":["python","requests"]},{"path":"/blog/answers/python-requests-exception-missingschema","type":"answer","title":"How to fix Python requests MissingSchema error?","description":"Python \"requests.MissingSchema\" exception is usually caused by a missing protocol part in the URL. Most commonly when relative URL is used.","tags":["python","requests"]},{"path":"/blog/answers/python-requests-exception-readtimeout","type":"answer","title":"How to fix Python requests ReadTimeout error?","description":"Python requests.ReadTimeout is caused when resources cannot be read fast enough. Here's how to fix it.","tags":["python","requests"]},{"path":"/blog/answers/python-requests-exception-sllerror","type":"answer","title":"How to fix Python requests SSLError?","description":"Python's requests.SSLError is caused when encryption certificates mismatch for HTTPS type of URLs. Here's how to fix it.","tags":["python","requests"]},{"path":"/blog/answers/python-requests-exception-toomanyredirects","type":"answer","title":"How to fix Python requests TooManyRedirects error?","description":"Python's requests.TooManyRedirects exception is raised when server continues to redirect >30 times. Here's how to fix it.","tags":["python","requests"]},{"path":"/blog/answers/python-requests-proxy-intro","type":"answer","title":"How to configure Python requests to use a proxy?","description":"Python requests supports many proxy types and options. Here's how to configure most proxy options for web scraping.","tags":["requests"]},{"path":"/blog/answers/selenium-chromedriver-in-path","type":"answer","title":"Selenium: chromedriver executable needs to be in PATH?","description":"selenium error \"chromedriver executable needs to be in PATH\" means that chrome driver is not installed or reachable - here's how to fix it.","tags":["python","selenium"]},{"path":"/blog/answers/selenium-geckodriver-in-path","type":"answer","title":"Selenium: geckodriver executable needs to be in PATH?","description":"selenium error \"geckodriver executable needs to be in PATH\" means that gecko driver is not installed or reachable - here's how to fix it.","tags":["python","selenium"]},{"path":"/blog/posts/how-to-rate-limit-asynchronous-python-requests","type":"article","title":"How to Rate Limit Async Requests in Python","description":"Tutorial on how to throttle asynchronous Python requests when web scraping. How to control web scraping speed to avoid being blocked.","tags":["blocking","python"]},{"path":"/blog/answers/403-status-code","type":"answer","title":"Web scraping - what is HTTP 403 status code?","description":"http error 403 means the client is being forbidden from acessing the requested resources. This could mean blocking or invalid request details.","tags":["blocking"]},{"path":"/blog/answers/429-status-code","type":"answer","title":"Web scraping - what is HTTP 429 status code?","description":"Response error code 429 means the client is making too many requests in a given time span and should slow down. Here's how to avoid it.","tags":["blocking"]},{"path":"/blog/answers/444-status-code","type":"answer","title":"What is 444 status code and how to avoid it?","description":"Response error code 444 means the server has unexpectedly closed connection. This could mean the web scraper is being blocked.","tags":["blocking"]},{"path":"/blog/answers/499-status-code","type":"answer","title":"Web scraping - what is HTTP 499 status code?","description":"Response error 499 generally means the server has closed the connection unexpectedly. This could mean the client is being blocked. Here's how to fix it.","tags":["blocking"]},{"path":"/blog/answers/503-status-code","type":"answer","title":"Web scraping - what is HTTP 503 status code?","description":"Response error 503 generally means the server is temporarily unavailable however it could also mean blocking. Here's how to fix it.","tags":["blocking"]},{"path":"/blog/answers/520-status-code","type":"answer","title":"Web scraping - what is HTTP 520 status code?","description":"Response error 502 generally means the server cannot create a valid response. This could also mean the client is being blocked. Here's how to fix it.","tags":["blocking"]},{"path":"/blog/answers/cloudflare-error-1006-1007-1008-access-denied","type":"answer","title":"What are Cloudflare Errors 1006, 1007, 1008?","description":"Cloudflare is a popular anti web scraping service and errors 1006, 1007 and 1008 are popular web scraping blocking errors. Here's how to avoid them.","tags":["blocking"]},{"path":"/blog/answers/cloudflare-error-1009-access-denied-country-or-region-banned","type":"answer","title":"What is Cloudflare Error 1009?","description":"Cloudflare is a popular web scraping blocking service and error 1009 access denied is a popular error for web scraper blocking. Here's how to avoid it.","tags":["blocking"]},{"path":"/blog/answers/cloudflare-error-1010-browser-signature","type":"answer","title":"What is Cloudflare Error 1010?","description":"Cloudflare is a popular web scraping blocking service and error 1010 access denied is a popular error for web scraper blocking. Here's how to avoid it.","tags":["blocking"]},{"path":"/blog/answers/cloudflare-error-1020-access-denied","type":"answer","title":"What is Cloudflare Error 1020?","description":"Cloudflare error 1020 access denied is a common web error when web scraping caused by Cloudflare anti scraping service. Here's how to avoid it.","tags":["blocking"]},{"path":"/blog/answers/how-to-install-requests-python","type":"answer","title":"3 ways to install Python Requests library","description":"Python requests library is a popular HTTP client and here's how to install it using pip, poetry and pipenv.","tags":["requests"]},{"path":"/blog/answers/perimeterx-verify-press-and-hold","type":"answer","title":"How to scrape Perimeter X: Please verify you are human?","description":"Perimeter X is a popular anti-scraping protection service - here's how to avoid it when web scraping.","tags":["blocking"]},{"path":"/blog/answers/xpath-vs-css-selectors","type":"answer","title":"XPath vs CSS selectors: what's the difference?","description":"CSS selectors and XPath are both path languages for HTML parsing. Xpath is more powerful but CSS is more approachable - which is one is better?","tags":["css-selectors","xpath"]},{"path":"/blog/posts/how-to-scrape-zoopla","type":"article","title":"How to Scrape Zoopla Real Estate Property Data in Python","description":"Tutorial for scraping Zoopla.com property data: houses for rent, sale, agent contact details and how to crawl search and sitemap pages.","tags":["python","real-estate","scrapeguide"]},{"path":"/blog/posts/parse-json-jmespath-python","type":"article","title":"Quick Intro to Parsing JSON with JMESPath in Python","description":"Tutorial on JMESPath - JSON query languge that is often used in web scraping to parse JSON datasets for scrape data. Plus an example project.","tags":["data-parsing","python"]},{"path":"/blog/answers/save-and-load-cookies-in-requests-python","type":"answer","title":"How to save and load cookies in Python requests?","description":"To save session between script runs we can save and load requests session cookies to disk. Here's how to do in Python requests.","tags":["http","python","requests"]},{"path":"/blog/posts/how-to-scrape-redfin","type":"article","title":"How to Scrape Redfin Real Estate Property Data in Python","description":"Tutorial on how to scrape Redfin.com for real estate property data using Python. How to extract property datasets for free and avoid blocking.","tags":["python","real-estate","scrapeguide"]},{"path":"/blog/answers/how-to-download-file-with-playwright","type":"answer","title":"How to download a file with Playwright and Python?","description":"To download files using Playwright we can either simulate the button click or extract the url and download it using HTTP. Here's how.","tags":["playwright"]},{"path":"/blog/answers/how-to-get-url-filetype-in-python","type":"answer","title":"How to get file type of an URL in Python?","description":"There are 2 ways to determine URL file type: guess by url extension using mimetypes module or do a HTTP HEAD request. Here's how.","tags":["crawling","python"]},{"path":"/blog/answers/how-to-load-local-files-in-playwright","type":"answer","title":"How to load local files in Playwright?","description":"To load local files as page URLs in Playwright we can use the file:// protocol. Here's how to do it.","tags":["playwright"]},{"path":"/blog/answers/how-to-save-and-load-cookies-in-playwright","type":"answer","title":"How to save and load cookies in Playwright?","description":"To persist playwright connection session between program runs we can save and load cookies to/from disk. Here's how.","tags":["playwright"]},{"path":"/blog/answers/how-to-take-screenshot-with-playwright","type":"answer","title":"How to take a screenshot with Playwright?","description":"To take page screenshots in playwright we can use page.screenshot() method. Here's how to select areas and how to screenshot them in playwright.","tags":["playwright","python"]},{"path":"/blog/posts/how-to-scrape-real-estate-property-data-using-python","type":"article","title":"How to Scrape Real Estate Property Data using Python","description":"Intro to web scraping real estate property data with Python. How to scrape popular targets like ZIllow, Realtor and dozens of tohers for free.","tags":["python","real-estate","scrapeguide"]},{"path":"/blog/answers/how-to-block-image-loading-in-selenium","type":"answer","title":"How to block image loading in Selenium?","description":"To increase Selenium's performance we can block images. To do that with Chrome browser \"prefs\" launch option can be used. Here's how.","tags":["python","selenium"]},{"path":"/blog/answers/scrapy-vs-beautifulsoup","type":"answer","title":"Scrapy vs BeautifulSoup: Which Should You Use?","description":"Compare Scrapy and Beautiful Soup by crawling, parsing, concurrency, retries, and project setup, with matched Python examples and a clear decision.","tags":["beautifulsoup","scrapy"]},{"path":"/blog/answers/scroll-to-element-selenium","type":"answer","title":"How to scroll to an element in Selenium?","description":"In Selenium, the scrollIntoView JavaScript function can be used to scroll to a specific HTML element. Here's how to use it in Selenium.","tags":["python","selenium"]},{"path":"/blog/posts/how-to-scrape-idealista","type":"article","title":"How to Scrape Idealista.com","description":"Intro to scraping Idealista.com real estate property data using Python. Hands on idealista scraper, parser and how to avoid blocking.","tags":["python","real-estate","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-realtorcom","type":"article","title":"How to Scrape Realtor.com - Real Estate Property Data","description":"Hands on tutorial on scraping Realtor.com using Python. How to scrape property information, pricing and track real-time updates and changes.","tags":["python","real-estate","scrapeguide"]},{"path":"/blog/answers/how-to-find-elements-by-xpath-in-playwright","type":"answer","title":"How to find elements by XPath selectors in Playwright?","description":"To execute XPath selectors in playwright the page.locator() method can be used. Here's how.","tags":["playwright","xpath"]},{"path":"/blog/answers/how-to-block-resources-in-playwright","type":"answer","title":"How to block resources in Playwright and Python?","description":"Blocking non-vital resources can drastically speed up Playwright. To do that page interception feature can be used. Here's how.","tags":["playwright","python"]},{"path":"/blog/answers/how-to-capture-xhr-requests-playwright","type":"answer","title":"How to capture background requests and responses in Playwright?","description":"To capture background requests and response in Playwright we can use request/response interception feature through page.on() method. Here's how.","tags":["playwright","python"]},{"path":"/blog/answers/how-to-find-elements-by-css-selectors-in-playwright","type":"answer","title":"How to find elements by CSS selectors in Playwright?","description":"To execute CSS selectors on current HTML data in Playwright the page.locator() method can be used. Here's how.","tags":["playwright","python"]},{"path":"/blog/answers/how-to-parse-dynamic-classes","type":"answer","title":"How to parse dynamic CSS classes when web scraping?","description":"Dynamic CSS can make be very difficult to scrape. There are a few tricks and common idioms to approach this though.","tags":["data-parsing"]},{"path":"/blog/answers/how-to-wait-for-page-to-load-in-playwright","type":"answer","title":"How to wait for page to load in Playwright?","description":"Playwright waits explained: wait_for_selector, wait_for_timeout and load states, plus what Timeout 30000ms exceeded means and how to fix it.","tags":["playwright","python"]},{"path":"/blog/answers/how-to-capture-xhr-requests-puppeteer","type":"answer","title":"How to capture background requests and responses in Puppeteer?","description":"To capture background requests and response in Puppeteer we can use page.on() method to intercept every request/response. Here's how.","tags":["puppeteer"]},{"path":"/blog/answers/how-to-find-html-elements-by-text-with-cheerio","type":"answer","title":"How to find HTML elements by text with Cheerio and NodeJS?","description":"To find HTML elements by text in NodeJS we can use cheerio library and special \":contains()\" selectors. Here's how to do it.","tags":["nodejs"]},{"path":"/blog/answers/how-to-ignore-non-html-urls-when-web-crawling","type":"answer","title":"How to ignore non HTML URLs when web crawling?","description":"When web crawling to avoid non-html pages we can test for page extensions or content types using HEAD requests. Here's how to do it.","tags":["crawling"]},{"path":"/blog/answers/how-to-select-elements-by-text-using-css-selectors","type":"answer","title":"How to select HTML elements by text using CSS Selectors?","description":"It's not possible to select HTML elements by text in original CSS selectors specification but here are some alternative ways to do it.","tags":["css-selectors","data-parsing"]},{"path":"/blog/answers/how-to-turn-html-to-text-in-python","type":"answer","title":"How to turn HTML to text in Python?","description":"To turn HTML data to text in Python we can use BeautifulSoup's get_text() method which strips away HTML data and leaves text as is. Here's how.","tags":["beautifulsoup","data-parsing"]},{"path":"/blog/answers/how-to-use-css-selectors-in-nodejs","type":"answer","title":"How to use CSS selectors in NodeJS when web scraping?","description":"There are many ways to execute CSS selectors on HTML text in NodeJS but cheerio and osmosis libraries are the most popular ones. Here's how to use them.","tags":["css-selectors","data-parsing","nodejs"]},{"path":"/blog/answers/how-to-use-css-selectors-in-python","type":"answer","title":"How to use CSS Selectors in Python?","description":"To parse HTML using CSS selectors in Python we can use either BeautifulSoup or Parsel packages. Here's how.","tags":["css-selectors","python"]},{"path":"/blog/answers/how-to-use-xpath-selectors-in-nodejs","type":"answer","title":"How to use XPath selectors in NodeJS when web scraping?","description":"To parse HTML using XPath in Nodejs we can use one of two popular libraries like osmosis or xmldom. Here's how.","tags":["data-parsing","nodejs","xpath"]},{"path":"/blog/answers/how-to-use-xpath-selectors-in-python","type":"answer","title":"How to use XPath selectors in Python?","description":"Python has several options for executing XPath selectors against HTML. The most popular ones are lxml and parsel. Here's how to use them.","tags":["data-parsing","python","xpath"]},{"path":"/blog/answers/why-cant-scraper-see-content","type":"answer","title":"Scraper doesn't see the data I see in the browser - why?","description":"This means that scraper is not rendereding javascript that is changing the page contents. To verify this disable javascript in your browser.","tags":["data-parsing","headless-browser"]},{"path":"/blog/answers/how-to-find-elements-by-css-selectors-in-puppeteer","type":"answer","title":"How to find elements by CSS selector in Puppeteer?","description":"To find HTML elements using CSS selectors in Puppeteer the $ and $eval methods can be used. Here's how to use them.","tags":["data-parsing","headless-browser","puppeteer"]},{"path":"/blog/answers/how-to-find-elements-by-xpath-in-puppeteer","type":"answer","title":"How to find elements by XPath in Puppeteer?","description":"To find elements by XPath using Puppeteer the \"$x()\" method can be used which will execute XPath selection on the current page DOM.","tags":["data-parsing","headless-browser","puppeteer","xpath"]},{"path":"/blog/answers/how-to-get-page-source-in-puppeteer","type":"answer","title":"How to get page source in Puppeteer?","description":"To retreive page source in Puppteer the page.content() method can be used. Here's how to use it and what are the possible options.","tags":["headless-browser","puppeteer","python"]},{"path":"/blog/answers/how-to-load-local-files-in-puppeteer","type":"answer","title":"How to load local files in Puppeteer?","description":"To load local files in Puppeteer the file:// URL protocol can be used as the URL protocol prefix which will load file from the file path URI","tags":["headless-browser","puppeteer"]},{"path":"/blog/answers/how-to-save-and-load-cookies-in-puppeteer","type":"answer","title":"How to save and load cookies in Puppeteer?","description":"To save and load cookies in Puppeteer page.setCookies() and page.cookies() methods can be used. Here's how to do it.","tags":["headless-browser","puppeteer"]},{"path":"/blog/answers/how-to-select-elements-by-class-in-xpath","type":"answer","title":"How to select elements by class in XPath?","description":"To select HTML elements by class name in XPath we can use the @ attribute selector and comparison function contains(). Here's how to do it.","tags":["data-parsing","xpath"]},{"path":"/blog/answers/how-to-select-elements-by-text-in-xpath","type":"answer","title":"How to select elements by text in XPath?","description":"To select elements by text using XPath, the contains() function can be used or re:test for selecting based on regular expression patterns.","tags":["data-parsing","xpath"]},{"path":"/blog/answers/how-to-take-screenshot-with-puppeteer","type":"answer","title":"How to take a screenshot with Puppeteer?","description":"Learn how to take Puppeteer screenshots in NodeJS. You will also learn how to customize it through resolution and viewport customization.","tags":["headless-browser","puppeteer"]},{"path":"/blog/answers/how-to-wait-for-page-to-load-in-puppeteer","type":"answer","title":"How to wait for a page to load in Puppeteer?","description":"To wait for a page to load in Puppeteer the best approach is to wait for a specific element to appear using page.waitForSelector() method. Here's how to do it.","tags":["headless-browser","puppeteer"]},{"path":"/blog/answers/how-to-find-elements-by-css-selectors-in-selenium","type":"answer","title":"How to find elements by CSS selector in Selenium","description":"To select HTML elements by CSS selectors in Selenium the driver.find_element() method can be used with the By.CSS_SELECTOR option. Here's how to do it.","tags":["css-selectors","data-parsing","selenium"]},{"path":"/blog/answers/how-to-find-elements-by-xpath-in-selenium","type":"answer","title":"How to find elements by XPath in Selenium","description":"To select HTML elements by CSS selectors in Selenium the driver.find_element() method can be used with the By.XPATH option. Here's how to do it.","tags":["headless-browser","python","selenium"]},{"path":"/blog/answers/how-to-find-elements-without-attribute-in-beautifulsoup","type":"answer","title":"How to find elements without a specific attribute in BeautifulSoup?","description":"To find HTML elements that do NOT contains a specific attribute we can use regular expression matching or lambda functions. Here's how to do it.","tags":["beautifulsoup","data-parsing","python"]},{"path":"/blog/answers/how-to-find-html-elements-by-multiple-tags-with-beautifulsoup","type":"answer","title":"How to find HTML elements by multiple tags with BeautifulSoup?","description":"To find HTML elements by one of many different element names we can use list of tags in find() methods or CSS selectors. Here's how to do it.","tags":["beautifulsoup","css-selectors","data-parsing"]},{"path":"/blog/answers/how-to-find-html-elements-by-text-with-beautifulsoup","type":"answer","title":"How to find HTML elements by text value with BeautifulSoup","description":"To find HTML elements by text value using Beautifulsoup and Python, regular expression patterns can be used in the text parameter of find functions. Here's how.","tags":[]},{"path":"/blog/answers/how-to-find-siblings-nodes-with-beautifulsoup","type":"answer","title":"How to find sibling HTML nodes using BeautifulSoup and Python?","description":"To find sibling HTML element nodes using BeautifulSoup the find_next_sibling() method can be used or CSS selector ~. Here's how to do it in Python.","tags":["beautifulsoup","css-selectors","data-parsing"]},{"path":"/blog/answers/how-to-get-page-source-in-selenium","type":"answer","title":"How to get page source in Selenium?","description":"To get full web page source in Selenium the driver.page_source property can be used. Here's how to do it in Python and Selenium.","tags":["headless-browser","python","selenium"]},{"path":"/blog/answers/how-to-save-and-load-cookies-in-selenium","type":"answer","title":"How to save and load cookies in Selenium?","description":"To save and load cookies of a Selenium browser we can use driver.get_cookies() and driver.add_cookies() methods. Here's how to use them.","tags":["headless-browser","python","selenium"]},{"path":"/blog/answers/how-to-select-values-between-two-elements-in-beautifulsoup","type":"answer","title":"How to select values between two nodes in BeautifulSoup and Python?","description":"To select HTML element located between two HTML elements using BeautifulSoup the find_next_sibling() method can be used. Here's how to do it.","tags":["beautifulsoup","data-parsing"]},{"path":"/blog/answers/how-to-take-screenshot-with-selenium","type":"answer","title":"How to take a screenshot with Selenium?","description":"To take a web page screenshot using Selenium the driver.save_screenshot() method can be used or element.screenshot() for specific element. Here's how to do it.","tags":["headless-browser","python","selenium"]},{"path":"/blog/answers/how-to-wait-for-page-to-load-in-selenium","type":"answer","title":"How to wait for page to load in Selenium?","description":"To wait for specific HTML element to load in Selenium the WebDriverWait() object can be used with presence_of_element_located parameters. Here's how to do it.","tags":["data-parsing","headless-browser","selenium"]},{"path":"/blog/answers/can-i-use-xpath-selectors-in-beautifulsoup","type":"answer","title":"Can I used XPath selectors in BeautifulSoup?","description":"BeautilfulSoup for Python doesn't support XPath selectors but there are popular alternatives to fill in this niche. Here are some.","tags":["beautifulsoup","data-parsing","xpath"]},{"path":"/blog/answers/how-to-find-all-links-using-beautifulsoup","type":"answer","title":"How to find all links using BeautifulSoup and Python?","description":"To find all links in the HTML pages using BeautifulSoup and Python the find_all() method can be used. Here's how to do it.","tags":["beautifulsoup","crawling","data-parsing"]},{"path":"/blog/answers/how-to-find-html-elements-by-attribute-with-beautifulsoup","type":"answer","title":"How to find HTML elements by attribute using BeautifulSoup?","description":"To find HTML node by a specific attribute value in BeautifulSoup the attribute match parameter can be used in the find() methods. Here's how.","tags":["beautifulsoup","css-selectors","data-parsing"]},{"path":"/blog/answers/how-to-find-html-elements-by-class","type":"answer","title":"How to find HTML elements by class?","description":"To find HTML nodes by class name CSS selectors or XPath can be used. For that .class css selector can be used or XPath's text() matcher.","tags":["css-selectors","data-parsing","xpath"]},{"path":"/blog/answers/how-to-find-html-elements-by-class-with-beautifulsoup","type":"answer","title":"How to find HTML element by class with BeautifulSoup?","description":"To find HTML node by class name using BeautifulSoup the class match parameter can be used using the find() methods. Here's how to do it.","tags":["beautifulsoup","css-selectors","data-parsing"]},{"path":"/blog/answers/how-to-scrape-tables-with-beautifulsoup","type":"answer","title":"How to scrape tables with BeautifulSoup?","description":"Scrape HTML tables with BeautifulSoup in Python, handling thead/tbody headers and row-header cells, plus when pandas.read_html or rendering is faster.","tags":["beautifulsoup","data-parsing"]},{"path":"/blog/answers/what-are-some-beautifulsoup-alternatives","type":"answer","title":"What are some BeautifulSoup alternatives in Python?","description":"BeautifulSoup is a popular HTML library for Python. It's most popular alternatives are lxml, parsel and html5lib. Here's how they differ from bs4.","tags":["beautifulsoup","css-selectors","data-parsing","python","xpath"]},{"path":"/blog/answers/whats-the-difference-between-scraping-and-crawling","type":"answer","title":"What's the difference between Web Scraping and Crawling?","description":"Web Scraping and Web Crawling are similar but not quite the same. Crawling is a form of web scraping and here are some major differences.","tags":["crawling"]},{"path":"/blog/answers/how-to-block-resources-in-puppeteer","type":"answer","title":"How to block resources in Puppeteer?","description":"Blocking non-critical resources in Puppeteer can drastically speed up the program. Here's how to do in Puppeteer and Nodejs.","tags":["puppeteer"]},{"path":"/blog/posts/how-to-scrape-hidden-web-data","type":"article","title":"How to Scrape Hidden Web Data","description":"Introduction to scraping data that is not visible in the HTML of the page. What is hidden web data and how to scrape it using Python.","tags":["data-parsing","python"]},{"path":"/blog/posts/how-to-ensure-web-scrapped-data-quality","type":"article","title":"How to Ensure Web Scrapped Data Quality","description":"Introduction to two tools used in web scrapped data quality validation - Cerberus and Pydantic. Why do we need to validate scraped data and how?","tags":["data-parsing","python"]},{"path":"/blog/answers/how-to-download-file-with-puppeteer","type":"answer","title":"How to download a file with Puppeteer?","description":"To download a file using Puppeteer and NodeJS we can either simulate the click on the download button or use HTTP client. Here's how to do it.","tags":["puppeteer"]},{"path":"/blog/posts/how-to-turn-web-scrapers-into-data-apis","type":"article","title":"How to Turn Web Scrapers into Data APIs","description":"Tutorial on how to create data APIs that scrape data on demand using Python and FastAPI. Real example project, best practices and tips.","tags":["api","project","python"]},{"path":"/blog/posts/how-to-scrape-glassdoor","type":"article","title":"How to Scrape Glassdoor (2026 update)","description":"Practical python web scraping tutorial for Glassdoor job listings, company data and reviews, salary information and other public data fields.","tags":["python","scrapeguide"]},{"path":"/blog/posts/web-scraping-with-playwright-and-python","type":"article","title":"Web Scraping With Playwright in 2026: A Python Guide","description":"Scrape JavaScript-rendered pages with Playwright and Python in 2026: locators, waiting, request interception, resource blocking, and concurrency.","tags":["headless-browser","python"]},{"path":"/blog/posts/how-to-rotate-proxies-in-web-scraping","type":"article","title":"How to Rotate Proxies in Web Scraping","description":"Introduction to proxy rotation in web scraping. What's the best way to rotate proxies for highest success and lowest blocking rates.","tags":["proxies","python"]},{"path":"/blog/posts/web-scraping-speed","type":"article","title":"Web Scraping Speed: Processes, Threads and Async","description":"How can we speed up web scraping using native Python technologies: processes, threads and asyncio. What's the best way to scale a web scraper?","tags":["python","scaling"]},{"path":"/blog/posts/how-to-scrape-indeedcom","type":"article","title":"How to Scrape Indeed.com (2026 Update)","description":"Tutorial on web scraping job listing data from Indeed.com using Python. How to collect and parse recruitment data without being blocked.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-algolia-search","type":"article","title":"How to Scrape Algolia Search","description":"Tutorial on web scraping Algolia search API which is used by many websites as a content search system.","tags":["hidden-api","project","python"]},{"path":"/blog/posts/crawling-with-python","type":"article","title":"How to Crawl the Web with Python","description":"Introduction to web crawling with Python. What is crawling, how it differs from scraping, deep dive into code and an example project.","tags":["python"]},{"path":"/blog/posts/how-to-scrape-zoominfo","type":"article","title":"How to Scrape Zoominfo Company Data (2026 Update)","description":"Practical tutorial on how to web scrape public company and people data from Zoominfo.com using Python and how to avoid being blocked.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-google-maps","type":"article","title":"How to Scrape Google Maps","description":"Tutorial on how to scrape Google Maps using browsers through Selenium, Playwright or ScrapFly's browser automation toolkits.","tags":["playwright","python","scrapeguide","selenium"]},{"path":"/blog/posts/how-to-scrape-wellfound-aka-angellist","type":"article","title":"How to Scrape Wellfound Company Data and Job Listings","description":"Tutorial for web scraping Wellfound.com (previously AngelList) company and job data using Python and  how to scrape without being blocked.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-crunchbase","type":"article","title":"How to Scrape Crunchbase in 2025","description":"Tutorial on how to scrape crunchbase.com business and related data using Python. How to avoid blocking to scrape data at scale and other tips.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-yellowpages","type":"article","title":"How to Scrape YellowPages.com in 2026","description":"Tutorial on how to scrape yellowpages.com business and review data using Python. How to avoid blocking to scrape data at scale and other tips.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-amazon","type":"article","title":"How to Scrape Amazon.com Product Data and Reviews","description":"Tutorial on how to scrape Amazon.com's product and review data using Python and how to avoid blocking and solve common Amazon scraping challenges.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-zillow","type":"article","title":"How to Scrape Zillow Real Estate Property Data in Python","description":"Tutorial on how to scrape Zillow.com for real estate property data using Python. How to extract datasets for free and avoid blocking.","tags":["python","real-estate","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-tripadvisor","type":"article","title":"How to Scrape TripAdvisor.com (2026 Updated)","description":"Step by step tutorial on how to scrape TripAdvisor.com reviews, hotel pricing and information data as well as other public details.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-aliexpress","type":"article","title":"How to Scrape Aliexpress.com (2026 Update)","description":"Tutorial on how to scrape Aliexpress.com product, review and pricing data using Python. How to avoid blocking to scrape at scale and other tips.","tags":["python","scrapeguide"]},{"path":"/blog/posts/search-engine-using-web-scraping","type":"article","title":"Creating Search Engine for any Website using Web Scraping","description":"Guide for creating a search engine for any website using web scraping in Python. How to crawl data, index it and display it via js powered GUI.","tags":["crawling","data-parsing","seo"]},{"path":"/blog/posts/how-to-scrape-bookingcom","type":"article","title":"How to Scrape Booking.com (2026 Update)","description":"Tutorial on how to scrape booking.com hotel and pricing data using Python. How to avoid blocking to web scrape data at scale and other tips.","tags":["python","scrapeguide"]},{"path":"/blog/posts/web-scraping-with-node-unblocker","type":"article","title":"Web Scraping With Node-Unblocker","description":"Tutorial on using Node-Unblocker - a nodejs library - to avoid blocking while web scraping and using it to optimize web scraping stacks.","tags":["blocking","nodejs","proxies"]},{"path":"/blog/posts/how-to-scrape-instagram","type":"article","title":"How to Scrape Instagram in 2026","description":"Tutorial on how to scrape instagram.com user and post data using pure Python. How to scrape instagram without loging in or being blocked.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-walmartcom","type":"article","title":"How to Scrape Walmart.com Product Data (2026 Update)","description":"Tutorial on how to scrape walmart.com product and review data using Python. How to avoid blocking to web scrape data at scale  and other tips.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-scrape-yelpcom","type":"article","title":"How to Scrape Yelp.com (2026 update)","description":"Tutorial on how to scrape yelp.com business and review data using Python. How to avoid blocking to web scrape data at scale  and other tips.","tags":["python","scrapeguide"]},{"path":"/blog/posts/how-to-avoid-web-scraping-blocking-headers","type":"article","title":"How Headers Are Used to Block Web Scrapers and How to Fix It","description":"Introduction to web scraping headers - what do they mean, how to configure them in web scrapers and how to avoid being blocked.","tags":["blocking","http"]},{"path":"/blog/posts/how-to-avoid-web-scraping-blocking-ip-addresses","type":"article","title":"How to Avoid Web Scraper IP Blocking?","description":"Introduction to how IP and proxy fingerprinting and analysis is used to block web scrapers and how to avoid these blocking techniques.","tags":["blocking","http","proxies"]},{"path":"/blog/posts/how-to-avoid-web-scraping-blocking-javascript","type":"article","title":"How Javascript is Used to Block Web Scrapers? In-Depth Guide","description":"Introduction to how javascript is used to detect web scrapers. What's in javascript fingerprint and how to correctly spoof it for web scraping.","tags":["blocking","headless-browser"]},{"path":"/blog/posts/how-to-avoid-web-scraping-blocking-tls","type":"article","title":"How TLS Fingerprint is Used to Block Web Scrapers?","description":"How TLS fingerprint (JA3) is being used to block web scrapers. In depth introduction and tips how to avoid this type of blocking.","tags":["blocking"]},{"path":"/blog/posts/how-to-scrape-without-getting-blocked-tutorial","type":"article","title":"5 Tools to Scrape Without Blocking and How it All Works","description":"Intro to how web scraper blocking works: Javascript, TLS and HTTP fingerprinting, IP Addresses and Proxies and how 5 popular tools can help.","tags":["blocking"]},{"path":"/blog/posts/web-scraping-graphql-with-python","type":"article","title":"Web Scraping Graphql with Python","description":"Introduction to web scraping graphql powered websites. How to create graphql queries in python and what are some common challenges.","tags":["graphql","http","python"]},{"path":"/blog/posts/web-scraping-with-python","type":"article","title":"Web Scraping with Python","description":"Tutorial on scraping in Python. Intro to HTTP clients and HTML parsing, common scraping challenges, idioms and hands-on example project.","tags":["data-parsing","http","python","scrapeguide"]},{"path":"/blog/posts/web-scraping-with-r","type":"article","title":"Web Scraping With R Tutorial and Example Project","description":"Tutorial on web scraping with R language. How to handle http connections, parse html files, best practices, tips and an example project.","tags":["data-parsing","http","r","scrapeguide"]},{"path":"/blog/posts/top-4-mobile-proxy-providers-for-web-scraping","type":"article","title":"Top 3 Mobile Proxy Providers for Web Scraping","description":"Analysis and comparison of top mobile proxy providers for web scraping and how to choose the right one to avoid web scraper blocking","tags":["blocking","proxies"]},{"path":"/blog/posts/introduction-to-proxies-in-web-scraping","type":"article","title":"The Complete Guide To Using Proxies For Web Scraping","description":"Introduction to proxy usage in web scraping. What types of proxies are there? How to evaluate proxy providers and avoid common issues.","tags":["blocking","proxies","scrapeguide"]},{"path":"/blog/posts/web-scraping-with-ruby","type":"article","title":"6 Best Ruby Web Scraping Libraries and APIs in 2026 (With a Working Scraper)","description":"Compare 6 Ruby web scraping libraries and APIs for HTTP, parsing, sessions, JavaScript, crawling, and blocked pages, then build a working scraper.","tags":["data-parsing","ruby","tools"]},{"path":"/blog/posts/web-scraping-with-nodejs","type":"article","title":"Web Scraping With NodeJS and Javascript","description":"Introduction to web scraping in Javascript through a real hands-on example. Scraping HTML, parsing it and common challenge overview.","tags":["css-selectors","data-parsing","http","nodejs","scrapeguide"]},{"path":"/blog/posts/web-scraping-with-puppeteer-and-nodejs","type":"article","title":"How to Web Scrape with Puppeteer and NodeJS in 2026","description":"Puppeteer and nodejs tutorial (javascript) for web scraping dynamic web pages and web apps. Tips and tricks, best practices and example project.","tags":["data-parsing","frameworks","headless-browser","nodejs","puppeteer"]},{"path":"/blog/posts/parsing-html-with-css","type":"article","title":"Parsing HTML with CSS Selectors","description":"Introduction to using CSS selectors to parse web-scraped content. Best practices, available tools and common challenges  by interactive examples.","tags":["css-selectors","data-parsing"]},{"path":"/blog/posts/parsing-html-with-xpath","type":"article","title":"Parsing HTML with Xpath","description":"Introduction to xpath in the context of web-scraping. How to extract data from HTML documents using xpath, best practices and available tools.","tags":["data-parsing","parsel","python","xpath"]},{"path":"/blog/posts/web-scraping-with-php-101","type":"article","title":"Web Scraping With PHP 101","description":"Introduction to web scraping with PHP. How to handle http connections, parse html files for data, best practices, tips and an example project.","tags":["data-parsing","http","php","scrapeguide","xpath"]},{"path":"/blog/posts/web-scraping-with-scrapy","type":"article","title":"Web Scraping With Scrapy: The Complete Guide in 2026","description":"Build and run a Scrapy 2.18 project with async start(), pagination, selector tests, item validation, pipelines, and JSON Lines export.","tags":["frameworks","python","scrapeguide","scrapy","xpath"]},{"path":"/blog/posts/web-scraping-with-selenium-and-python","type":"article","title":"Web Scraping with Selenium and Python","description":"Selenium and Python tutorial for web scraping dynamic, javascript powered websites using a headless Chrome webdriver. Real life example project.","tags":["headless-browser","python","selenium"]},{"path":"/blog/posts/web-scraping-with-python-beautifulsoup","type":"article","title":"How to Parse Web Data with Python and Beautifulsoup","description":"Introduction to web scraping with Python and BeautifulSoup HTML parsing library used in scraping. How to find text in scraped web data.","tags":["beautifulsoup","data-parsing","python"]},{"path":"/blog/posts/scraping-using-browsers","type":"article","title":"How to Scrape Dynamic Websites Using Headless Web Browsers","description":"Intro to using headless web browser and libraries like Puppeteer, Playwright and Selenium in web scraping dynamic websites.","tags":["headless-browser","playwright","puppeteer","python","selenium"]}]}