# Scrapfly Blog Tutorials, guides, and insights on web scraping, data extraction, and automation 287 articles Search articles [ ](https://scrapfly.io/blog/feed.xml "Subscribe via RSS") [All](https://scrapfly.io/blog) [ai](https://scrapfly.io/blog/tag/ai) [api](https://scrapfly.io/blog/tag/api) [automation](https://scrapfly.io/blog/tag/automation) [beautifulsoup](https://scrapfly.io/blog/tag/beautifulsoup) [blocking](https://scrapfly.io/blog/tag/blocking) [cloud-browser](https://scrapfly.io/blog/tag/cloud-browser) [crawling](https://scrapfly.io/blog/tag/crawling) [css-selectors](https://scrapfly.io/blog/tag/css-selectors) [curl](https://scrapfly.io/blog/tag/curl) [data-parsing](https://scrapfly.io/blog/tag/data-parsing) [ecommerce](https://scrapfly.io/blog/tag/ecommerce) [fashion](https://scrapfly.io/blog/tag/fashion) [frameworks](https://scrapfly.io/blog/tag/frameworks) [golang](https://scrapfly.io/blog/tag/golang) [graphql](https://scrapfly.io/blog/tag/graphql) [headless-browser](https://scrapfly.io/blog/tag/headless-browser) [hidden-api](https://scrapfly.io/blog/tag/hidden-api) [http](https://scrapfly.io/blog/tag/http) [httpx](https://scrapfly.io/blog/tag/httpx) [java](https://scrapfly.io/blog/tag/java) [javascript](https://scrapfly.io/blog/tag/javascript) [jupyter](https://scrapfly.io/blog/tag/jupyter) [nodejs](https://scrapfly.io/blog/tag/nodejs) [parsel](https://scrapfly.io/blog/tag/parsel) [php](https://scrapfly.io/blog/tag/php) [playwright](https://scrapfly.io/blog/tag/playwright) [project](https://scrapfly.io/blog/tag/project) [proxies](https://scrapfly.io/blog/tag/proxies) [puppeteer](https://scrapfly.io/blog/tag/puppeteer) [python](https://scrapfly.io/blog/tag/python) [r](https://scrapfly.io/blog/tag/r) [real-estate](https://scrapfly.io/blog/tag/real-estate) [requests](https://scrapfly.io/blog/tag/requests) [ruby](https://scrapfly.io/blog/tag/ruby) [scaling](https://scrapfly.io/blog/tag/scaling) [scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [scrapy](https://scrapfly.io/blog/tag/scrapy) [screenshots](https://scrapfly.io/blog/tag/screenshots) [selenium](https://scrapfly.io/blog/tag/selenium) [seo](https://scrapfly.io/blog/tag/seo) [tools](https://scrapfly.io/blog/tag/tools) [typescript](https://scrapfly.io/blog/tag/typescript) [web-scraping](https://scrapfly.io/blog/tag/web-scraping) [xpath](https://scrapfly.io/blog/tag/xpath) ## // Articles 12 posts [ Latest python blocking api ### How to Scrape Air France Flights with Python in 2026 Scrape Air France round-trip flight offers with Python and the Scrapfly Cloud Browser API: walk the booking widget, capture the GraphQL booking response, and return structured records with price, time... Jun 23, 2026 25 min read ](https://scrapfly.io/blog/posts/how-to-scrape-air-france-flights) [ python scrapeguide ### Web Scraping for Lead Generation: Build Your Own B2B Database Learn how to build a B2B lead database by scraping public sources. Covers source discovery, extraction without getting ... Jun 23, 2026 28 min read ](https://scrapfly.io/blog/posts/how-to-scrape-leads) [ python ecommerce project ### How to Build a Grocery Price Comparison Tool with Python Build a grocery price comparison tool that scrapes Walmart, Instacart, and Kroger prices by zip code, matches products, ... Jun 23, 2026 19 min read ](https://scrapfly.io/blog/posts/how-to-build-a-grocery-price-comparison-tool-with-python) [ python api ai ### Web Scraping for AI Agents in 2026 How AI agents consume the web, why their fetch layer breaks, and how to build agent-grade web access that holds up in pr... Jun 23, 2026 23 min read ](https://scrapfly.io/blog/posts/ai-agent-web-scraping) [ python web-scraping ### The Best Open-Source Social Media Scrapers for 2026 Ranked guide to the best open-source social media scraping tools in 2026. Eight maintained scrapers for Instagram, X, Li... Jun 23, 2026 16 min read ](https://scrapfly.io/blog/posts/best-social-media-scraping-tools) [ ### 11 Best Anti-Bot Bypass Tools for Web Scraping in 2026 Compare the best anti-bot bypass tools for web scraping, from managed APIs to open-source stealth browsers. See which to... Jun 23, 2026 22 min read ](https://scrapfly.io/blog/posts/best-anti-bot-bypass-tools) [ python api ai ### Best AI Web Scraping Tools for LLM and RAG Pipelines in 2026 A by-job ranking of the best AI web scraping tools for 2026, from prompt-based extraction to MCP servers and open-source... Jun 23, 2026 17 min read ](https://scrapfly.io/blog/posts/best-tools-for-ai-webscraping) [ api data-parsing ### 10 Best Public Data Sources for Lead Generation in 2026 A ranked directory of 10 public data sources for B2B lead generation, with the fields, access method, and freshness of e... Jun 19, 2026 19 min read ](https://scrapfly.io/blog/posts/best-public-data-sources-for-lead-generation) [ python data-parsing scrapeguide ### How to Scrape an Entire Product Catalogue with Python Learn how to discover, crawl, and extract every product from an e-commerce catalog in Python, then keep that data fresh ... Jun 17, 2026 17 min read ](https://scrapfly.io/blog/posts/how-to-scrape-large-product-catalogs) [ python ai ### How to Scrape ChatGPT Responses in 2026 Scrape ChatGPT responses with Python: when to pick the OpenAI API, how to handle Cloudflare and streaming responses, and... Jun 16, 2026 20 min read ](https://scrapfly.io/blog/posts/how-to-scrape-chatgpt) [ python ai ### How to Build a Web Scraping Agent with Claude Learn how to build a reliable web scraping agent with Claude. Covers Claude Code skills, the Anthropic API, autonomous a... Jun 16, 2026 21 min read ](https://scrapfly.io/blog/posts/how-to-build-a-web-scraping-agent-with-claude) [ python api ai ### How to Build a Web Scraping Agent with Gemini Build a Gemini web scraping agent that works on real sites. Covers Gemini CLI skills, URL Context limits, Python pipelin... Jun 16, 2026 21 min read ](https://scrapfly.io/blog/posts/gemini-for-webscraping) 1 [2](https://scrapfly.io/blog?page=2) [3](https://scrapfly.io/blog?page=3) … [24](https://scrapfly.io/blog?page=24) [ ](https://scrapfly.io/blog?page=2) Page 1 of 24 · 287 articles ## ? Quick Answers ### nodejs [View all ](https://scrapfly.io/blog/answers) - [ Q How to take screenshots in NodeJS? ](https://scrapfly.io/blog/answers/how-to-take-screenshots-nodejs) ### css-selectors [View all ](https://scrapfly.io/blog/answers) - [ Q How to use CSS Selectors in Nim ? ](https://scrapfly.io/blog/answers/how-to-use-css-selectors-in-nim) - [ Q How to select elements by attribute using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-attribute-containing-value-css-selectors) - [ Q How to select elements by class using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-class-css-selectors) - [ Q How to select elements by ID using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-id-css-selectors) - [ Q How to select following siblings using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-following-sibling-element-css-selectors) ### proxies [View all ](https://scrapfly.io/blog/answers) - [ Q How to Solve the cURL (60) Error When Using Proxy? ](https://scrapfly.io/blog/answers/how-to-solve-the-curl-60-error-when-proxy) - [ Q How To Use Proxy With cURL? ](https://scrapfly.io/blog/answers/how-to-use-proxy-with-curl) - [ Q What is The cURL (28) Error, Couldn't connect to server? ](https://scrapfly.io/blog/answers/what-is-the-curl-28-error) - [ Q Mobile vs Residential Proxies - which to choose for scraping? ](https://scrapfly.io/blog/answers/mobile-vs-residential-proxies-whats-the-difference) - [ Q What are private proxies and how are they used in scraping? ](https://scrapfly.io/blog/answers/what-are-private-proxies-compared-to-shared) ### curl [View all ](https://scrapfly.io/blog/answers) - [ Q How To Download a File With cURL? ](https://scrapfly.io/blog/answers/how-to-download-file-curl) - [ Q How to Follow Redirects In cURL? ](https://scrapfly.io/blog/answers/how-to-follow-redirects-in-curl) - [ Q How To Send cURL POST Requests? ](https://scrapfly.io/blog/answers/how-to-send-a-post-request-using-curl) - [ Q How to Send a HEAD Request With cURL? ](https://scrapfly.io/blog/answers/how-to-send-curl-head-requests) - [ Q How To Send Multiple cURL Requests in Parallel? ](https://scrapfly.io/blog/answers/how-to-send-multiple-curl-requests-in-parallel) ### http [View all ](https://scrapfly.io/blog/answers) - [ Q How to Copy as cURL With Brave? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-brave) - [ Q How To Copy as cURL With Google Chrome? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-chrome) - [ Q How to Copy as cURL With Edge? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-edge) - [ Q How to Copy as cURL With Firefox? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-firefox) - [ Q How to Copy as cURL With Safari? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-safari) ### tools [View all ](https://scrapfly.io/blog/answers) - [ Q How to edit Local Storage data using browser Devtools ](https://scrapfly.io/blog/answers/how-to-edit-local-storage-using-devtools) - [ Q How to edit cookies in Chrome devtools? ](https://scrapfly.io/blog/answers/how-to-edit-cookies-using-chrome-devtools) ### python [View all ](https://scrapfly.io/blog/answers) - [ Q How to scrape HTML table to Excel Spreadsheet (.xlsx)? ](https://scrapfly.io/blog/answers/html-table-to-xlsx-python-beautifulsoup) - [ Q How to handle popup dialogs in Playwright? ](https://scrapfly.io/blog/answers/how-to-click-on-alert-dialog-in-playwright) - [ Q How to scrape images from a website? ](https://scrapfly.io/blog/answers/how-to-scrape-images-from-website) - [ Q How to check if element exists in Playwright? ](https://scrapfly.io/blog/answers/how-to-check-for-element-in-playwright) - [ Q How to select dictionary key recursively in Python? ](https://scrapfly.io/blog/answers/how-to-select-dictionary-key-recursively-in-python) ### puppeteer [View all ](https://scrapfly.io/blog/answers) - [ Q How to handle popup dialogs in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-click-on-alert-dialog-in-puppeteer) - [ Q How to click on cookie popups and modal alerts in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-puppeteer) - [ Q How to scroll to the bottom of the page with Puppeteer? ](https://scrapfly.io/blog/answers/how-to-scroll-to-the-bottom-with-puppeteer) - [ Q Getting started with Puppeteer Stealth ](https://scrapfly.io/blog/answers/how-to-use-puppeteer-stealth-what-does-it-do) ### selenium [View all ](https://scrapfly.io/blog/answers) - [ Q How to handle popup dialogs in Selenium? ](https://scrapfly.io/blog/answers/how-to-click-on-alert-dialog-in-selenium) - [ Q How to click on cookie popups and modal alerts in Selenium? ](https://scrapfly.io/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-selenium) - [ Q How to block resources in Selenium and Python? ](https://scrapfly.io/blog/answers/how-to-block-resources-in-selenium) - [ Q How to capture background requests and responses in Selenium? ](https://scrapfly.io/blog/answers/how-to-capture-xhr-requests-selenium) - [ Q How to scroll to the bottom of the page with Selenium? ](https://scrapfly.io/blog/answers/how-to-scroll-to-the-bottom-with-selenium) ### playwright [View all ](https://scrapfly.io/blog/answers) - [ Q How to click on cookie popups and modal alerts in Playwright? ](https://scrapfly.io/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-playwright) - [ Q How to scroll to the bottom of the page with Playwright? ](https://scrapfly.io/blog/answers/how-to-scroll-to-the-bottom-with-playwright) - [ Q How to run Playwright in Jupyter notebooks? ](https://scrapfly.io/blog/answers/playwright-in-ipython) ### xpath [View all ](https://scrapfly.io/blog/answers) - [ Q How to select elements by attribute value in XPath? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-attribute-value) - [ Q How to count selections in XPath and why? ](https://scrapfly.io/blog/answers/how-to-count-selectors-in-xpath-and-why) - [ Q How to get the name of an HTML element in XPath? ](https://scrapfly.io/blog/answers/how-to-get-name-of-selected-element-in-xpath) - [ Q How to join values using XPath concat? ](https://scrapfly.io/blog/answers/how-to-join-values-in-xpath) - [ Q How to reverse expressions in XPath? ](https://scrapfly.io/blog/answers/how-to-reverse-expression-in-xpath) ### scrapy [View all ](https://scrapfly.io/blog/answers) - [ Q What are scrapy middlewares and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-middlewares-and-how-to-use-them) - [ Q What are scrapy pipelines and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-pipelines-and-how-to-use-them) - [ Q How to pass custom parameters to scrapy spiders? ](https://scrapfly.io/blog/answers/how-to-pass-parameters-to-scrapy-spiders-cli) - [ Q How to pass data between scrapy callbacks in Scrapy? ](https://scrapfly.io/blog/answers/how-to-pass-data-between-scrapy-callbacks) - [ Q How to pass data from start\_requests to parse callbacks in scrapy? ](https://scrapfly.io/blog/answers/how-to-pass-data-from-start-request-to-callbacks-scrapy) ### headless-browser [View all ](https://scrapfly.io/blog/answers) - [ Q How to use headless browsers with scrapy? ](https://scrapfly.io/blog/answers/how-to-use-headless-browsers-with-scrapy) ### data-parsing [View all ](https://scrapfly.io/blog/answers) - [ Q How to select last element in XPath? ](https://scrapfly.io/blog/answers/how-to-select-last-element-in-xpath) - [ Q How to select all elements between two elements in XPath? ](https://scrapfly.io/blog/answers/how-to-select-all-elements-between-two-known-elements-in-xpath) ### requests [View all ](https://scrapfly.io/blog/answers) - [ Q How to configure Python requests to use a proxy? ](https://scrapfly.io/blog/answers/python-requests-proxy-intro) ### blocking [View all ](https://scrapfly.io/blog/answers) - [ Q Web scraping - what is HTTP 403 status code? ](https://scrapfly.io/blog/answers/403-status-code) - [ Q Web scraping - what is HTTP 429 status code? ](https://scrapfly.io/blog/answers/429-status-code) - [ Q What is 444 status code and how to avoid it? ](https://scrapfly.io/blog/answers/444-status-code) - [ Q Web scraping - what is HTTP 499 status code? ](https://scrapfly.io/blog/answers/499-status-code) - [ Q Web scraping - what is HTTP 503 status code? ](https://scrapfly.io/blog/answers/503-status-code) ## Ready to scale your web scraping? Anti-bot bypass, browser rendering, and rotating proxies, all in one API. [ Try Scrapfly for FREE ](https://scrapfly.io/register) [ View Documentation ](https://scrapfly.io/docs/onboarding)