# Scrapfly Blog Tutorials, guides, and insights on web scraping, data extraction, and automation 375 articles Search articles [ ](https://scrapfly.io/blog/feed.xml "Subscribe via RSS") [All](https://scrapfly.io/blog) [ai](https://scrapfly.io/blog/tag/ai) [anti-bot](https://scrapfly.io/blog/tag/anti-bot) [api](https://scrapfly.io/blog/tag/api) [beautifulsoup](https://scrapfly.io/blog/tag/beautifulsoup) [blocking](https://scrapfly.io/blog/tag/blocking) [cloud-browser](https://scrapfly.io/blog/tag/cloud-browser) [crawling](https://scrapfly.io/blog/tag/crawling) [css-selectors](https://scrapfly.io/blog/tag/css-selectors) [curl](https://scrapfly.io/blog/tag/curl) [data](https://scrapfly.io/blog/tag/data) [data-parsing](https://scrapfly.io/blog/tag/data-parsing) [debugging](https://scrapfly.io/blog/tag/debugging) [ecommerce](https://scrapfly.io/blog/tag/ecommerce) [fashion](https://scrapfly.io/blog/tag/fashion) [frameworks](https://scrapfly.io/blog/tag/frameworks) [golang](https://scrapfly.io/blog/tag/golang) [graphql](https://scrapfly.io/blog/tag/graphql) [headless-browser](https://scrapfly.io/blog/tag/headless-browser) [hidden-api](https://scrapfly.io/blog/tag/hidden-api) [http](https://scrapfly.io/blog/tag/http) [httpx](https://scrapfly.io/blog/tag/httpx) [java](https://scrapfly.io/blog/tag/java) [javascript](https://scrapfly.io/blog/tag/javascript) [jupyter](https://scrapfly.io/blog/tag/jupyter) [nodejs](https://scrapfly.io/blog/tag/nodejs) [parsel](https://scrapfly.io/blog/tag/parsel) [php](https://scrapfly.io/blog/tag/php) [playwright](https://scrapfly.io/blog/tag/playwright) [project](https://scrapfly.io/blog/tag/project) [proxies](https://scrapfly.io/blog/tag/proxies) [puppeteer](https://scrapfly.io/blog/tag/puppeteer) [python](https://scrapfly.io/blog/tag/python) [r](https://scrapfly.io/blog/tag/r) [real-estate](https://scrapfly.io/blog/tag/real-estate) [requests](https://scrapfly.io/blog/tag/requests) [ruby](https://scrapfly.io/blog/tag/ruby) [rust](https://scrapfly.io/blog/tag/rust) [scaling](https://scrapfly.io/blog/tag/scaling) [scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [scrapy](https://scrapfly.io/blog/tag/scrapy) [screenshots](https://scrapfly.io/blog/tag/screenshots) [selenium](https://scrapfly.io/blog/tag/selenium) [seo](https://scrapfly.io/blog/tag/seo) [tools](https://scrapfly.io/blog/tag/tools) [typescript](https://scrapfly.io/blog/tag/typescript) [web-scraping](https://scrapfly.io/blog/tag/web-scraping) [wget](https://scrapfly.io/blog/tag/wget) [xpath](https://scrapfly.io/blog/tag/xpath) ## // Articles 12 posts [ Latest python proxies selenium ### How to Use Proxies with Selenium: Authentication, Rotation, and Residential Proxies A tested walkthrough of Python Selenium proxies: why Chrome drops credentials, the auth methods that still work, rotation, residential gateways and geo checks. Oct 19, 2026 33 min read ](https://scrapfly.io/blog/posts/how-to-use-proxies-with-selenium) [ python scrapeguide ### How to Scrape Shopify Stores for Products, Prices, and Variants Build a resilient Shopify storefront scraper around products.json, JSON-LD, response classification, and measured fallba... Oct 19, 2026 23 min read ](https://scrapfly.io/blog/posts/how-to-scrape-shopify-stores) [ http proxies frameworks ### How to Use a Proxy With C# HttpClient in 2026 How to configure authenticated and rotating proxies with C# HttpClient and SocketsHttpHandler, including the connection-... Oct 16, 2026 13 min read ](https://scrapfly.io/blog/posts/csharp-httpclient-proxy) [ python data-parsing scrapeguide ### How to Test a Web Scraper Before Production Build parser tests, HTML fixtures, live checks, and deployment gates that catch silent web scraper failures before they ... Oct 16, 2026 24 min read ](https://scrapfly.io/blog/posts/how-to-test-web-scraper-before-production) [ python data-parsing hidden-api ### How to Scrape idealo: Product Offers, Prices, and Price History Scrape idealo product pages for the full merchant offer list, prices, and price history with Python and the Scrapfly Web... Oct 16, 2026 24 min read ](https://scrapfly.io/blog/posts/how-to-scrape-idealo) [ tools web-scraping rust ### 6 Top Rust Web Scraping Libraries and APIs in 2026 Six current Rust options for managed page access, static HTML, article extraction, and browser control. Oct 16, 2026 12 min read ](https://scrapfly.io/blog/posts/best-rust-web-scraping-libraries) [ tools data-parsing golang ### 9 Best Go (Golang) Web Scraping Libraries and APIs in 2026 Nine Go web scraping libraries and APIs compared by the job each does best, current release, and license. Oct 14, 2026 14 min read ](https://scrapfly.io/blog/posts/best-go-web-scraping-libraries) [ headless-browser puppeteer cloud-browser ### 7 Best Puppeteer Alternatives for Web Scraping in 2026 A job-by-job guide to replacing Puppeteer for web scraping, covering library rewrites, crawl frameworks, no-browser rout... Oct 14, 2026 23 min read ](https://scrapfly.io/blog/posts/best-puppeteer-alternatives-for-web-scraping) [ python hidden-api scrapeguide ### How to Scrape Pinterest in 2026 Learn to scrape Pinterest pins, boards, profiles, and images with Python by starting a session, calling the hidden resou... Oct 08, 2026 16 min read ](https://scrapfly.io/blog/posts/how-to-scrape-pinterest) [ python hidden-api scrapeguide ### How to Scrape Goodreads for Book Data, Ratings, and Reviews A practical guide to scraping Goodreads book pages, ratings, and review samples in Python, reading JSON-LD and the em... Sep 29, 2026 26 min read ](https://scrapfly.io/blog/posts/how-to-scrape-goodreads) [ python scrapeguide ### How to Scrape News Articles from Any Website Learn how to scrape news articles from any website in Python, extracting headline, author, date, and body with Newspaper... Sep 29, 2026 19 min read ](https://scrapfly.io/blog/posts/how-to-scrape-news-articles) [ python scrapeguide ### Top 4 (Twitter) X Scrapers on GitHub That Still Work in 2026 Four surviving X scraper repositories compared by coverage, account requirements, maintenance, and infrastructure burden... Sep 17, 2026 18 min read ](https://scrapfly.io/blog/posts/best-twitter-scrapers-github) 1 [2](https://scrapfly.io/blog?page=2) [3](https://scrapfly.io/blog?page=3) … [32](https://scrapfly.io/blog?page=32) [ ](https://scrapfly.io/blog?page=2) Page 1 of 32 · 375 articles ## ? Quick Answers ### http [View all ](https://scrapfly.io/blog/answers) - [ Q How to Persist Sessions Across Scrapy Requests ](https://scrapfly.io/blog/answers/persist-sessions-across-scrapy-requests) - [ Q How to Copy as cURL With Safari? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-safari) - [ Q How to Copy as cURL With Firefox? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-firefox) - [ Q How to Copy as cURL With Edge? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-edge) - [ Q How To Copy as cURL With Google Chrome? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-chrome) ### nodejs [View all ](https://scrapfly.io/blog/answers) - [ Q How to take screenshots in NodeJS? ](https://scrapfly.io/blog/answers/how-to-take-screenshots-nodejs) ### css-selectors [View all ](https://scrapfly.io/blog/answers) - [ Q How to use CSS Selectors in Nim ? ](https://scrapfly.io/blog/answers/how-to-use-css-selectors-in-nim) - [ Q Is it possible to select preceding siblings using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-preceding-sibling-element-css-selectors) - [ Q How to select following siblings using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-following-sibling-element-css-selectors) - [ Q How to select elements by ID using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-id-css-selectors) - [ Q How to select elements by class using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-class-css-selectors) ### proxies [View all ](https://scrapfly.io/blog/answers) - [ Q What is The cURL (28) Error, Couldn't connect to server? ](https://scrapfly.io/blog/answers/what-is-the-curl-28-error) - [ Q How To Use Proxy With cURL? ](https://scrapfly.io/blog/answers/how-to-use-proxy-with-curl) - [ Q How to Solve the cURL (60) Error When Using Proxy? ](https://scrapfly.io/blog/answers/how-to-solve-the-curl-60-error-when-proxy) - [ Q What are SOCKS5 proxies and how they compare to HTTP proxies? ](https://scrapfly.io/blog/answers/what-are-socks5-proxies-in-web-scraping) - [ Q What are private proxies and how are they used in scraping? ](https://scrapfly.io/blog/answers/what-are-private-proxies-compared-to-shared) ### curl [View all ](https://scrapfly.io/blog/answers) - [ Q How to Set User Agent With cURL? ](https://scrapfly.io/blog/answers/how-to-set-curl-user-agent) - [ Q How to Use cURL Config Files? ](https://scrapfly.io/blog/answers/how-to-set-curl-config-file) - [ Q How to Set cURL Authentication - Full Examples Guide ](https://scrapfly.io/blog/answers/how-to-set-authorization-with-curl-full-examples-guide) - [ Q How To Send Multiple cURL Requests in Parallel? ](https://scrapfly.io/blog/answers/how-to-send-multiple-curl-requests-in-parallel) - [ Q How to Send a HEAD Request With cURL? ](https://scrapfly.io/blog/answers/how-to-send-curl-head-requests) ### python [View all ](https://scrapfly.io/blog/answers) - [ Q How to scrape HTML table to Excel Spreadsheet (.xlsx)? ](https://scrapfly.io/blog/answers/html-table-to-xlsx-python-beautifulsoup) - [ Q How to handle popup dialogs in Playwright? ](https://scrapfly.io/blog/answers/how-to-click-on-alert-dialog-in-playwright) - [ Q How to scrape images from a website? ](https://scrapfly.io/blog/answers/how-to-scrape-images-from-website) - [ Q What are some ways to parse JSON datasets in Python? ](https://scrapfly.io/blog/answers/what-are-some-ways-to-parse-json-datasets-in-python) - [ Q How to select dictionary key recursively in Python? ](https://scrapfly.io/blog/answers/how-to-select-dictionary-key-recursively-in-python) ### tools [View all ](https://scrapfly.io/blog/answers) - [ Q How to edit Local Storage data using browser Devtools ](https://scrapfly.io/blog/answers/how-to-edit-local-storage-using-devtools) - [ Q How to edit cookies in Chrome devtools? ](https://scrapfly.io/blog/answers/how-to-edit-cookies-using-chrome-devtools) ### selenium [View all ](https://scrapfly.io/blog/answers) - [ Q How to click on cookie popups and modal alerts in Selenium? ](https://scrapfly.io/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-selenium) - [ Q How to handle popup dialogs in Selenium? ](https://scrapfly.io/blog/answers/how-to-click-on-alert-dialog-in-selenium) - [ Q How to scroll to the bottom of the page with Selenium? ](https://scrapfly.io/blog/answers/how-to-scroll-to-the-bottom-with-selenium) - [ Q How to capture background requests and responses in Selenium? ](https://scrapfly.io/blog/answers/how-to-capture-xhr-requests-selenium) - [ Q How to block resources in Selenium and Python? ](https://scrapfly.io/blog/answers/how-to-block-resources-in-selenium) ### puppeteer [View all ](https://scrapfly.io/blog/answers) - [ Q How to click on cookie popups and modal alerts in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-puppeteer) - [ Q How to handle popup dialogs in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-click-on-alert-dialog-in-puppeteer) - [ Q How to scroll to the bottom of the page with Puppeteer? ](https://scrapfly.io/blog/answers/how-to-scroll-to-the-bottom-with-puppeteer) - [ Q Getting started with Puppeteer Stealth ](https://scrapfly.io/blog/answers/how-to-use-puppeteer-stealth-what-does-it-do) ### playwright [View all ](https://scrapfly.io/blog/answers) - [ Q How to click on cookie popups and modal alerts in Playwright? ](https://scrapfly.io/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-playwright) - [ Q How to scroll to the bottom of the page with Playwright? ](https://scrapfly.io/blog/answers/how-to-scroll-to-the-bottom-with-playwright) - [ Q How to run Playwright in Jupyter notebooks? ](https://scrapfly.io/blog/answers/playwright-in-ipython) ### scrapy [View all ](https://scrapfly.io/blog/answers) - [ Q What are scrapy middlewares and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-middlewares-and-how-to-use-them) - [ Q What are scrapy pipelines and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-pipelines-and-how-to-use-them) - [ Q How to pass custom parameters to scrapy spiders? ](https://scrapfly.io/blog/answers/how-to-pass-parameters-to-scrapy-spiders-cli) - [ Q What are scrapy Item and ItemLoader objects and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-items-and-itemloaders) - [ Q How to pass data from start\_requests to parse callbacks in scrapy? ](https://scrapfly.io/blog/answers/how-to-pass-data-from-start-request-to-callbacks-scrapy) ### xpath [View all ](https://scrapfly.io/blog/answers) - [ Q How to select elements by attribute value in XPath? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-attribute-value) - [ Q How to select sibling elements in XPath? ](https://scrapfly.io/blog/answers/how-to-select-sibling-elements-using-xpath) - [ Q How to select elements of a specific position in XPath? ](https://scrapfly.io/blog/answers/how-to-select-elements-of-specific-position-in-xpath) - [ Q How to select any element using wildcard in XPath? ](https://scrapfly.io/blog/answers/how-to-select-elements-of-any-name-using-wildcards-in-xpath) - [ Q How to select elements by ID in XPath? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-id-in-xpath) ### headless-browser [View all ](https://scrapfly.io/blog/answers) - [ Q How to use headless browsers with scrapy? ](https://scrapfly.io/blog/answers/how-to-use-headless-browsers-with-scrapy) ### data-parsing [View all ](https://scrapfly.io/blog/answers) - [ Q How to select last element in XPath? ](https://scrapfly.io/blog/answers/how-to-select-last-element-in-xpath) - [ Q How to select all elements between two elements in XPath? ](https://scrapfly.io/blog/answers/how-to-select-all-elements-between-two-known-elements-in-xpath) ### requests [View all ](https://scrapfly.io/blog/answers) - [ Q How to configure Python requests to use a proxy? ](https://scrapfly.io/blog/answers/python-requests-proxy-intro) ### blocking [View all ](https://scrapfly.io/blog/answers) - [ Q Web scraping - what is HTTP 520 status code? ](https://scrapfly.io/blog/answers/520-status-code) - [ Q Web scraping - what is HTTP 503 status code? ](https://scrapfly.io/blog/answers/503-status-code) - [ Q Web scraping - what is HTTP 499 status code? ](https://scrapfly.io/blog/answers/499-status-code) - [ Q What is 444 status code and how to avoid it? ](https://scrapfly.io/blog/answers/444-status-code) - [ Q Web scraping - what is HTTP 429 status code? ](https://scrapfly.io/blog/answers/429-status-code) ## Ready to scale your web scraping? Anti-bot bypass, browser rendering, and rotating proxies, all in one API. [ Try Scrapfly for FREE ](https://scrapfly.io/register) [ View Documentation ](https://scrapfly.io/docs/onboarding)