# Scrapfly Blog Tutorials, guides, and insights on web scraping, data extraction, and automation 366 articles Search articles [ ](https://scrapfly.io/blog/feed.xml "Subscribe via RSS") [All](https://scrapfly.io/blog) [ai](https://scrapfly.io/blog/tag/ai) [anti-bot](https://scrapfly.io/blog/tag/anti-bot) [api](https://scrapfly.io/blog/tag/api) [beautifulsoup](https://scrapfly.io/blog/tag/beautifulsoup) [blocking](https://scrapfly.io/blog/tag/blocking) [cloud-browser](https://scrapfly.io/blog/tag/cloud-browser) [crawling](https://scrapfly.io/blog/tag/crawling) [css-selectors](https://scrapfly.io/blog/tag/css-selectors) [curl](https://scrapfly.io/blog/tag/curl) [data](https://scrapfly.io/blog/tag/data) [data-parsing](https://scrapfly.io/blog/tag/data-parsing) [debugging](https://scrapfly.io/blog/tag/debugging) [ecommerce](https://scrapfly.io/blog/tag/ecommerce) [fashion](https://scrapfly.io/blog/tag/fashion) [frameworks](https://scrapfly.io/blog/tag/frameworks) [golang](https://scrapfly.io/blog/tag/golang) [graphql](https://scrapfly.io/blog/tag/graphql) [headless-browser](https://scrapfly.io/blog/tag/headless-browser) [hidden-api](https://scrapfly.io/blog/tag/hidden-api) [http](https://scrapfly.io/blog/tag/http) [httpx](https://scrapfly.io/blog/tag/httpx) [java](https://scrapfly.io/blog/tag/java) [javascript](https://scrapfly.io/blog/tag/javascript) [jupyter](https://scrapfly.io/blog/tag/jupyter) [nodejs](https://scrapfly.io/blog/tag/nodejs) [parsel](https://scrapfly.io/blog/tag/parsel) [php](https://scrapfly.io/blog/tag/php) [playwright](https://scrapfly.io/blog/tag/playwright) [project](https://scrapfly.io/blog/tag/project) [proxies](https://scrapfly.io/blog/tag/proxies) [puppeteer](https://scrapfly.io/blog/tag/puppeteer) [python](https://scrapfly.io/blog/tag/python) [r](https://scrapfly.io/blog/tag/r) [real-estate](https://scrapfly.io/blog/tag/real-estate) [requests](https://scrapfly.io/blog/tag/requests) [ruby](https://scrapfly.io/blog/tag/ruby) [scaling](https://scrapfly.io/blog/tag/scaling) [scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [scrapy](https://scrapfly.io/blog/tag/scrapy) [screenshots](https://scrapfly.io/blog/tag/screenshots) [selenium](https://scrapfly.io/blog/tag/selenium) [seo](https://scrapfly.io/blog/tag/seo) [tools](https://scrapfly.io/blog/tag/tools) [typescript](https://scrapfly.io/blog/tag/typescript) [web-scraping](https://scrapfly.io/blog/tag/web-scraping) [wget](https://scrapfly.io/blog/tag/wget) [xpath](https://scrapfly.io/blog/tag/xpath) ## // Articles 12 posts [ Latest python hidden-api scrapeguide ### How to Scrape Goodreads for Book Data, Ratings, and Reviews A practical guide to scraping Goodreads book pages, ratings, and review samples in Python, reading JSON-LD and the embedded Next.js Apollo state before falling back to CSS selectors, plus how to ... Sep 29, 2026 26 min read ](https://scrapfly.io/blog/posts/how-to-scrape-goodreads) [ python scrapeguide ### How to Scrape News Articles from Any Website Learn how to scrape news articles from any website in Python, extracting headline, author, date, and body with Newspaper... Sep 29, 2026 19 min read ](https://scrapfly.io/blog/posts/how-to-scrape-news-articles) [ python scrapeguide ### 4 X (Twitter) Scrapers on GitHub That Still Work in 2026 Four surviving X scraper repositories compared by coverage, account requirements, maintenance, and infrastructure burden... Sep 17, 2026 18 min read ](https://scrapfly.io/blog/posts/best-twitter-scrapers-github) [ python api scrapeguide ### 8 Best Financial Data APIs for Developers in 2026 Eight financial data APIs compared by workload, delivery method, history depth, entitlement, and what the licence actual... Sep 17, 2026 18 min read ](https://scrapfly.io/blog/posts/best-financial-data-apis) [ python blocking headless-browser ### How to Bypass F5 when Web Scraping in 2026 Tell BIG-IP Bot Defense from F5 Distributed Cloud Bot Defense, inspect its browser telemetry, and configure a browser-ba... Sep 17, 2026 10 min read ](https://scrapfly.io/blog/posts/how-to-bypass-f5-when-web-scraping) [ python api ### 6 Best Local Business Data APIs for Developers in 2026 Six local business data APIs compared by coverage and licensing, from platform place search and reviews to open map data... Sep 16, 2026 14 min read ](https://scrapfly.io/blog/posts/best-local-business-data-apis) [ python api data-parsing ### 8 Best Social Media Data APIs for Developers in 2026 Compare eight official social media APIs across posts, video, communities, research, professional data, and federated ne... Sep 16, 2026 18 min read ](https://scrapfly.io/blog/posts/best-social-media-data-apis) [ tools data-parsing frameworks ### 7 Best C# Web Scraping Libraries in 2026 A workflow based comparison of seven C# web scraping libraries across three layers, covering HTML parsing with Html Ag... Sep 16, 2026 20 min read ](https://scrapfly.io/blog/posts/best-csharp-web-scraping-libraries) [ tools data-parsing css-selectors ### HtmlAgilityPack vs AngleSharp: Which C# HTML Parser Should You Use? A fixture based comparison of Html Agility Pack and AngleSharp for C# HTML parsing, covering XPath versus CSS selectors,... Sep 11, 2026 16 min read ](https://scrapfly.io/blog/posts/htmlagilitypack-vs-anglesharp) [ python hidden-api scrapeguide ### How to Scrape DHL Tracking Status and Shipment Events A practical guide to scraping DHL parcel tracking status and shipment events in Python, covering Track & Trace page insp... Sep 11, 2026 19 min read ](https://scrapfly.io/blog/posts/how-to-scrape-dhl) [ python scrapeguide ### How to Scrape Goibibo Hotel and Flight Prices Learn how to scrape Goibibo hotel and flight prices with Python in 2026. Use Scrapfly ASP, India geo-targeting, and brow... Sep 11, 2026 18 min read ](https://scrapfly.io/blog/posts/how-to-scrape-goibibo-hotel-and-flight-prices) [ python data-parsing ### 5 Best Open-Source Amazon Scrapers in 2026 Five Amazon scraper repos on GitHub compared with rechecked stars, commit dates, licenses, code paths, and live failure ... Sep 11, 2026 17 min read ](https://scrapfly.io/blog/posts/best-amazon-scrapers-github) 1 [2](https://scrapfly.io/blog?page=2) [3](https://scrapfly.io/blog?page=3) … [31](https://scrapfly.io/blog?page=31) [ ](https://scrapfly.io/blog?page=2) Page 1 of 31 · 366 articles ## ? Quick Answers ### http [View all ](https://scrapfly.io/blog/answers) - [ Q How to Persist Sessions Across Scrapy Requests ](https://scrapfly.io/blog/answers/persist-sessions-across-scrapy-requests) - [ Q How to Copy as cURL With Safari? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-safari) - [ Q How to Copy as cURL With Firefox? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-firefox) - [ Q How to Copy as cURL With Edge? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-edge) - [ Q How To Copy as cURL With Google Chrome? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-chrome) ### nodejs [View all ](https://scrapfly.io/blog/answers) - [ Q How to take screenshots in NodeJS? ](https://scrapfly.io/blog/answers/how-to-take-screenshots-nodejs) ### css-selectors [View all ](https://scrapfly.io/blog/answers) - [ Q How to use CSS Selectors in Nim ? ](https://scrapfly.io/blog/answers/how-to-use-css-selectors-in-nim) - [ Q Is it possible to select preceding siblings using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-preceding-sibling-element-css-selectors) - [ Q How to select following siblings using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-following-sibling-element-css-selectors) - [ Q How to select elements by ID using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-id-css-selectors) - [ Q How to select elements by class using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-class-css-selectors) ### proxies [View all ](https://scrapfly.io/blog/answers) - [ Q What is The cURL (28) Error, Couldn't connect to server? ](https://scrapfly.io/blog/answers/what-is-the-curl-28-error) - [ Q How To Use Proxy With cURL? ](https://scrapfly.io/blog/answers/how-to-use-proxy-with-curl) - [ Q How to Solve the cURL (60) Error When Using Proxy? ](https://scrapfly.io/blog/answers/how-to-solve-the-curl-60-error-when-proxy) - [ Q What are SOCKS5 proxies and how they compare to HTTP proxies? ](https://scrapfly.io/blog/answers/what-are-socks5-proxies-in-web-scraping) - [ Q What are private proxies and how are they used in scraping? ](https://scrapfly.io/blog/answers/what-are-private-proxies-compared-to-shared) ### curl [View all ](https://scrapfly.io/blog/answers) - [ Q How to Set User Agent With cURL? ](https://scrapfly.io/blog/answers/how-to-set-curl-user-agent) - [ Q How to Use cURL Config Files? ](https://scrapfly.io/blog/answers/how-to-set-curl-config-file) - [ Q How to Set cURL Authentication - Full Examples Guide ](https://scrapfly.io/blog/answers/how-to-set-authorization-with-curl-full-examples-guide) - [ Q How To Send Multiple cURL Requests in Parallel? ](https://scrapfly.io/blog/answers/how-to-send-multiple-curl-requests-in-parallel) - [ Q How to Send a HEAD Request With cURL? ](https://scrapfly.io/blog/answers/how-to-send-curl-head-requests) ### python [View all ](https://scrapfly.io/blog/answers) - [ Q How to scrape HTML table to Excel Spreadsheet (.xlsx)? ](https://scrapfly.io/blog/answers/html-table-to-xlsx-python-beautifulsoup) - [ Q How to handle popup dialogs in Playwright? ](https://scrapfly.io/blog/answers/how-to-click-on-alert-dialog-in-playwright) - [ Q How to scrape images from a website? ](https://scrapfly.io/blog/answers/how-to-scrape-images-from-website) - [ Q What are some ways to parse JSON datasets in Python? ](https://scrapfly.io/blog/answers/what-are-some-ways-to-parse-json-datasets-in-python) - [ Q How to select dictionary key recursively in Python? ](https://scrapfly.io/blog/answers/how-to-select-dictionary-key-recursively-in-python) ### tools [View all ](https://scrapfly.io/blog/answers) - [ Q How to edit Local Storage data using browser Devtools ](https://scrapfly.io/blog/answers/how-to-edit-local-storage-using-devtools) - [ Q How to edit cookies in Chrome devtools? ](https://scrapfly.io/blog/answers/how-to-edit-cookies-using-chrome-devtools) ### selenium [View all ](https://scrapfly.io/blog/answers) - [ Q How to click on cookie popups and modal alerts in Selenium? ](https://scrapfly.io/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-selenium) - [ Q How to handle popup dialogs in Selenium? ](https://scrapfly.io/blog/answers/how-to-click-on-alert-dialog-in-selenium) - [ Q How to scroll to the bottom of the page with Selenium? ](https://scrapfly.io/blog/answers/how-to-scroll-to-the-bottom-with-selenium) - [ Q How to capture background requests and responses in Selenium? ](https://scrapfly.io/blog/answers/how-to-capture-xhr-requests-selenium) - [ Q How to block resources in Selenium and Python? ](https://scrapfly.io/blog/answers/how-to-block-resources-in-selenium) ### puppeteer [View all ](https://scrapfly.io/blog/answers) - [ Q How to click on cookie popups and modal alerts in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-puppeteer) - [ Q How to handle popup dialogs in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-click-on-alert-dialog-in-puppeteer) - [ Q How to scroll to the bottom of the page with Puppeteer? ](https://scrapfly.io/blog/answers/how-to-scroll-to-the-bottom-with-puppeteer) - [ Q Getting started with Puppeteer Stealth ](https://scrapfly.io/blog/answers/how-to-use-puppeteer-stealth-what-does-it-do) ### playwright [View all ](https://scrapfly.io/blog/answers) - [ Q How to click on cookie popups and modal alerts in Playwright? ](https://scrapfly.io/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-playwright) - [ Q How to scroll to the bottom of the page with Playwright? ](https://scrapfly.io/blog/answers/how-to-scroll-to-the-bottom-with-playwright) - [ Q How to run Playwright in Jupyter notebooks? ](https://scrapfly.io/blog/answers/playwright-in-ipython) ### scrapy [View all ](https://scrapfly.io/blog/answers) - [ Q What are scrapy middlewares and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-middlewares-and-how-to-use-them) - [ Q What are scrapy pipelines and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-pipelines-and-how-to-use-them) - [ Q How to pass custom parameters to scrapy spiders? ](https://scrapfly.io/blog/answers/how-to-pass-parameters-to-scrapy-spiders-cli) - [ Q What are scrapy Item and ItemLoader objects and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-items-and-itemloaders) - [ Q How to pass data from start\_requests to parse callbacks in scrapy? ](https://scrapfly.io/blog/answers/how-to-pass-data-from-start-request-to-callbacks-scrapy) ### xpath [View all ](https://scrapfly.io/blog/answers) - [ Q How to select elements by attribute value in XPath? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-attribute-value) - [ Q How to select sibling elements in XPath? ](https://scrapfly.io/blog/answers/how-to-select-sibling-elements-using-xpath) - [ Q How to select elements of a specific position in XPath? ](https://scrapfly.io/blog/answers/how-to-select-elements-of-specific-position-in-xpath) - [ Q How to select any element using wildcard in XPath? ](https://scrapfly.io/blog/answers/how-to-select-elements-of-any-name-using-wildcards-in-xpath) - [ Q How to select elements by ID in XPath? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-id-in-xpath) ### headless-browser [View all ](https://scrapfly.io/blog/answers) - [ Q How to use headless browsers with scrapy? ](https://scrapfly.io/blog/answers/how-to-use-headless-browsers-with-scrapy) ### data-parsing [View all ](https://scrapfly.io/blog/answers) - [ Q How to select last element in XPath? ](https://scrapfly.io/blog/answers/how-to-select-last-element-in-xpath) - [ Q How to select all elements between two elements in XPath? ](https://scrapfly.io/blog/answers/how-to-select-all-elements-between-two-known-elements-in-xpath) ### requests [View all ](https://scrapfly.io/blog/answers) - [ Q How to configure Python requests to use a proxy? ](https://scrapfly.io/blog/answers/python-requests-proxy-intro) ### blocking [View all ](https://scrapfly.io/blog/answers) - [ Q Web scraping - what is HTTP 520 status code? ](https://scrapfly.io/blog/answers/520-status-code) - [ Q Web scraping - what is HTTP 503 status code? ](https://scrapfly.io/blog/answers/503-status-code) - [ Q Web scraping - what is HTTP 499 status code? ](https://scrapfly.io/blog/answers/499-status-code) - [ Q What is 444 status code and how to avoid it? ](https://scrapfly.io/blog/answers/444-status-code) - [ Q Web scraping - what is HTTP 429 status code? ](https://scrapfly.io/blog/answers/429-status-code) ## Ready to scale your web scraping? Anti-bot bypass, browser rendering, and rotating proxies, all in one API. [ Try Scrapfly for FREE ](https://scrapfly.io/register) [ View Documentation ](https://scrapfly.io/docs/onboarding)