# Scrapfly Blog Tutorials, guides, and insights on web scraping, data extraction, and automation 342 articles Search articles [ ](https://scrapfly.io/blog/feed.xml "Subscribe via RSS") [All](https://scrapfly.io/blog) [ai](https://scrapfly.io/blog/tag/ai) [anti-bot](https://scrapfly.io/blog/tag/anti-bot) [api](https://scrapfly.io/blog/tag/api) [beautifulsoup](https://scrapfly.io/blog/tag/beautifulsoup) [blocking](https://scrapfly.io/blog/tag/blocking) [cloud-browser](https://scrapfly.io/blog/tag/cloud-browser) [crawling](https://scrapfly.io/blog/tag/crawling) [css-selectors](https://scrapfly.io/blog/tag/css-selectors) [curl](https://scrapfly.io/blog/tag/curl) [data-parsing](https://scrapfly.io/blog/tag/data-parsing) [ecommerce](https://scrapfly.io/blog/tag/ecommerce) [fashion](https://scrapfly.io/blog/tag/fashion) [frameworks](https://scrapfly.io/blog/tag/frameworks) [golang](https://scrapfly.io/blog/tag/golang) [graphql](https://scrapfly.io/blog/tag/graphql) [headless-browser](https://scrapfly.io/blog/tag/headless-browser) [hidden-api](https://scrapfly.io/blog/tag/hidden-api) [http](https://scrapfly.io/blog/tag/http) [httpx](https://scrapfly.io/blog/tag/httpx) [java](https://scrapfly.io/blog/tag/java) [javascript](https://scrapfly.io/blog/tag/javascript) [jupyter](https://scrapfly.io/blog/tag/jupyter) [nodejs](https://scrapfly.io/blog/tag/nodejs) [parsel](https://scrapfly.io/blog/tag/parsel) [php](https://scrapfly.io/blog/tag/php) [playwright](https://scrapfly.io/blog/tag/playwright) [project](https://scrapfly.io/blog/tag/project) [proxies](https://scrapfly.io/blog/tag/proxies) [puppeteer](https://scrapfly.io/blog/tag/puppeteer) [python](https://scrapfly.io/blog/tag/python) [r](https://scrapfly.io/blog/tag/r) [real-estate](https://scrapfly.io/blog/tag/real-estate) [requests](https://scrapfly.io/blog/tag/requests) [ruby](https://scrapfly.io/blog/tag/ruby) [scaling](https://scrapfly.io/blog/tag/scaling) [scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [scrapy](https://scrapfly.io/blog/tag/scrapy) [screenshots](https://scrapfly.io/blog/tag/screenshots) [selenium](https://scrapfly.io/blog/tag/selenium) [seo](https://scrapfly.io/blog/tag/seo) [tools](https://scrapfly.io/blog/tag/tools) [typescript](https://scrapfly.io/blog/tag/typescript) [web-scraping](https://scrapfly.io/blog/tag/web-scraping) [wget](https://scrapfly.io/blog/tag/wget) [xpath](https://scrapfly.io/blog/tag/xpath) ## // Articles 12 posts [ Latest python scrapeguide ### 6 Best Open-Source YouTube Scrapers in 2026 Six open-source YouTube scrapers ranked by job, with an August 11, 2026 GitHub snapshot and two projects that failed in our tests. Aug 24, 2026 22 min read ](https://scrapfly.io/blog/posts/best-open-source-youtube-scrapers) [ python hidden-api scrapeguide ### 3 Best Open-Source Job Scrapers for Multi-Board Data in 2026 Three open-source job scrapers ranked by board coverage, schema, and dated maintenance, plus a source-available Scrapfly... Aug 24, 2026 20 min read ](https://scrapfly.io/blog/posts/best-open-source-job-scrapers) [ python hidden-api scrapeguide ### How to Scrape Target.com Product and Pricing Data Extract Target.com product and pricing data from the internal Redsky API in 2026, handle store-keyed prices, and get pas... Aug 24, 2026 20 min read ](https://scrapfly.io/blog/posts/how-to-scrape-target-com) [ python headless-browser ai ### 12 Python Web Scraping Projects for 2026 12 Python web scraping projects laddered from static HTML to Scrapy crawls and AI agents, each with skills, a data sourc... Aug 24, 2026 15 min read ](https://scrapfly.io/blog/posts/python-web-scraping-projects-from-beginner-to-advanced) [ python blocking headless-browser ### How to Scrape Skyscanner Flight Prices in Python (2026) Learn how to scrape Skyscanner flight results in Python by building deep-link URLs, rendering the search, and capturing ... Aug 14, 2026 24 min read ](https://scrapfly.io/blog/posts/how-to-scrape-skyscanner) [ python ### How to Scrape Airbnb Listings and Prices (2026) Scrape Airbnb search results, listing details, fee-inclusive prices, guest reviews, and availability calendars with Pyth... Aug 14, 2026 22 min read ](https://scrapfly.io/blog/posts/how-to-scrape-airbnb) [ python scrapeguide ### 5 Best Open-Source LinkedIn Scrapers on GitHub in 2026 Five open-source LinkedIn scraper repos on GitHub ranked by auth model, dated maintenance and honest ban risk, with live... Aug 14, 2026 19 min read ](https://scrapfly.io/blog/posts/best-linkedin-scrapers-github) [ python scrapeguide ### How to Scrape Lowe's Product, Price, Search, and Store Location Data Scrape Lowe's product and pricing data from embedded page state, discover products and stores, and handle Akamai with Sc... Aug 13, 2026 14 min read ](https://scrapfly.io/blog/posts/how-to-scrape-lowe) [ python scrapeguide ### How to Scrape IMDb Data, Ratings, and Featured Reviews Fetch protected IMDb pages through Scrapfly ASP, then parse JSON-LD and \_\_NEXT\_DATA\_\_ for movie metadata, ratings, featu... Aug 13, 2026 16 min read ](https://scrapfly.io/blog/posts/how-to-scrape-imdb) [ python blocking api ### How to Scrape DigiKey Electronic Component Data in 2026 Scrape DigiKey pricing, stock, and parametric specs past its Cloudflare challenge, and learn when the official API v4 is... Aug 13, 2026 22 min read ](https://scrapfly.io/blog/posts/how-to-scrape-digikey) [ python scrapeguide ### 6 Best Open-Source Instagram Scrapers in 2026 A ranked comparison of the six best open-source Instagram scrapers in 2026 with honest notes on auth model, ban risk, an... Aug 11, 2026 17 min read ](https://scrapfly.io/blog/posts/best-open-source-instagram-scrapers) [ ai tools ### 5 MCP Servers for Web Scraping and Browser Automation in 2026 Compare five MCP servers by the work they can actually do: protected-site scraping, browser control, debugging, static f... Aug 11, 2026 16 min read ](https://scrapfly.io/blog/posts/best-mcp-servers-for-web-scraping) 1 [2](https://scrapfly.io/blog?page=2) [3](https://scrapfly.io/blog?page=3) … [29](https://scrapfly.io/blog?page=29) [ ](https://scrapfly.io/blog?page=2) Page 1 of 29 · 342 articles ## ? Quick Answers ### nodejs [View all ](https://scrapfly.io/blog/answers) - [ Q How to take screenshots in NodeJS? ](https://scrapfly.io/blog/answers/how-to-take-screenshots-nodejs) ### css-selectors [View all ](https://scrapfly.io/blog/answers) - [ Q How to use CSS Selectors in Nim ? ](https://scrapfly.io/blog/answers/how-to-use-css-selectors-in-nim) - [ Q How to select elements by attribute using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-attribute-containing-value-css-selectors) - [ Q How to select elements by class using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-class-css-selectors) - [ Q How to select elements by ID using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-id-css-selectors) - [ Q How to select following siblings using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-following-sibling-element-css-selectors) ### proxies [View all ](https://scrapfly.io/blog/answers) - [ Q How to Solve the cURL (60) Error When Using Proxy? ](https://scrapfly.io/blog/answers/how-to-solve-the-curl-60-error-when-proxy) - [ Q How To Use Proxy With cURL? ](https://scrapfly.io/blog/answers/how-to-use-proxy-with-curl) - [ Q What is The cURL (28) Error, Couldn't connect to server? ](https://scrapfly.io/blog/answers/what-is-the-curl-28-error) - [ Q Mobile vs Residential Proxies - which to choose for scraping? ](https://scrapfly.io/blog/answers/mobile-vs-residential-proxies-whats-the-difference) - [ Q What are private proxies and how are they used in scraping? ](https://scrapfly.io/blog/answers/what-are-private-proxies-compared-to-shared) ### curl [View all ](https://scrapfly.io/blog/answers) - [ Q How To Download a File With cURL? ](https://scrapfly.io/blog/answers/how-to-download-file-curl) - [ Q How to Follow Redirects In cURL? ](https://scrapfly.io/blog/answers/how-to-follow-redirects-in-curl) - [ Q How To Send cURL POST Requests? ](https://scrapfly.io/blog/answers/how-to-send-a-post-request-using-curl) - [ Q How to Send a HEAD Request With cURL? ](https://scrapfly.io/blog/answers/how-to-send-curl-head-requests) - [ Q How To Send Multiple cURL Requests in Parallel? ](https://scrapfly.io/blog/answers/how-to-send-multiple-curl-requests-in-parallel) ### http [View all ](https://scrapfly.io/blog/answers) - [ Q How to Copy as cURL With Brave? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-brave) - [ Q How To Copy as cURL With Google Chrome? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-chrome) - [ Q How to Copy as cURL With Edge? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-edge) - [ Q How to Copy as cURL With Firefox? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-firefox) - [ Q How to Copy as cURL With Safari? ](https://scrapfly.io/blog/answers/how-to-copy-as-curl-with-safari) ### tools [View all ](https://scrapfly.io/blog/answers) - [ Q How to edit Local Storage data using browser Devtools ](https://scrapfly.io/blog/answers/how-to-edit-local-storage-using-devtools) - [ Q How to edit cookies in Chrome devtools? ](https://scrapfly.io/blog/answers/how-to-edit-cookies-using-chrome-devtools) ### python [View all ](https://scrapfly.io/blog/answers) - [ Q How to scrape HTML table to Excel Spreadsheet (.xlsx)? ](https://scrapfly.io/blog/answers/html-table-to-xlsx-python-beautifulsoup) - [ Q How to handle popup dialogs in Playwright? ](https://scrapfly.io/blog/answers/how-to-click-on-alert-dialog-in-playwright) - [ Q How to scrape images from a website? ](https://scrapfly.io/blog/answers/how-to-scrape-images-from-website) - [ Q How to check if element exists in Playwright? ](https://scrapfly.io/blog/answers/how-to-check-for-element-in-playwright) - [ Q How to select dictionary key recursively in Python? ](https://scrapfly.io/blog/answers/how-to-select-dictionary-key-recursively-in-python) ### puppeteer [View all ](https://scrapfly.io/blog/answers) - [ Q How to handle popup dialogs in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-click-on-alert-dialog-in-puppeteer) - [ Q How to click on cookie popups and modal alerts in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-puppeteer) - [ Q How to scroll to the bottom of the page with Puppeteer? ](https://scrapfly.io/blog/answers/how-to-scroll-to-the-bottom-with-puppeteer) - [ Q Getting started with Puppeteer Stealth ](https://scrapfly.io/blog/answers/how-to-use-puppeteer-stealth-what-does-it-do) ### selenium [View all ](https://scrapfly.io/blog/answers) - [ Q How to handle popup dialogs in Selenium? ](https://scrapfly.io/blog/answers/how-to-click-on-alert-dialog-in-selenium) - [ Q How to click on cookie popups and modal alerts in Selenium? ](https://scrapfly.io/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-selenium) - [ Q How to block resources in Selenium and Python? ](https://scrapfly.io/blog/answers/how-to-block-resources-in-selenium) - [ Q How to capture background requests and responses in Selenium? ](https://scrapfly.io/blog/answers/how-to-capture-xhr-requests-selenium) - [ Q How to scroll to the bottom of the page with Selenium? ](https://scrapfly.io/blog/answers/how-to-scroll-to-the-bottom-with-selenium) ### playwright [View all ](https://scrapfly.io/blog/answers) - [ Q How to click on cookie popups and modal alerts in Playwright? ](https://scrapfly.io/blog/answers/how-to-click-on-modal-alerts-like-cookie-pop-up-in-playwright) - [ Q How to scroll to the bottom of the page with Playwright? ](https://scrapfly.io/blog/answers/how-to-scroll-to-the-bottom-with-playwright) - [ Q How to run Playwright in Jupyter notebooks? ](https://scrapfly.io/blog/answers/playwright-in-ipython) ### xpath [View all ](https://scrapfly.io/blog/answers) - [ Q How to select elements by attribute value in XPath? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-attribute-value) - [ Q How to count selections in XPath and why? ](https://scrapfly.io/blog/answers/how-to-count-selectors-in-xpath-and-why) - [ Q How to get the name of an HTML element in XPath? ](https://scrapfly.io/blog/answers/how-to-get-name-of-selected-element-in-xpath) - [ Q How to join values using XPath concat? ](https://scrapfly.io/blog/answers/how-to-join-values-in-xpath) - [ Q How to reverse expressions in XPath? ](https://scrapfly.io/blog/answers/how-to-reverse-expression-in-xpath) ### scrapy [View all ](https://scrapfly.io/blog/answers) - [ Q What are scrapy middlewares and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-middlewares-and-how-to-use-them) - [ Q What are scrapy pipelines and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-pipelines-and-how-to-use-them) - [ Q How to pass custom parameters to scrapy spiders? ](https://scrapfly.io/blog/answers/how-to-pass-parameters-to-scrapy-spiders-cli) - [ Q How to pass data between scrapy callbacks in Scrapy? ](https://scrapfly.io/blog/answers/how-to-pass-data-between-scrapy-callbacks) - [ Q How to pass data from start\_requests to parse callbacks in scrapy? ](https://scrapfly.io/blog/answers/how-to-pass-data-from-start-request-to-callbacks-scrapy) ### headless-browser [View all ](https://scrapfly.io/blog/answers) - [ Q How to use headless browsers with scrapy? ](https://scrapfly.io/blog/answers/how-to-use-headless-browsers-with-scrapy) ### data-parsing [View all ](https://scrapfly.io/blog/answers) - [ Q How to select last element in XPath? ](https://scrapfly.io/blog/answers/how-to-select-last-element-in-xpath) - [ Q How to select all elements between two elements in XPath? ](https://scrapfly.io/blog/answers/how-to-select-all-elements-between-two-known-elements-in-xpath) ### requests [View all ](https://scrapfly.io/blog/answers) - [ Q How to configure Python requests to use a proxy? ](https://scrapfly.io/blog/answers/python-requests-proxy-intro) ### blocking [View all ](https://scrapfly.io/blog/answers) - [ Q Web scraping - what is HTTP 403 status code? ](https://scrapfly.io/blog/answers/403-status-code) - [ Q Web scraping - what is HTTP 429 status code? ](https://scrapfly.io/blog/answers/429-status-code) - [ Q What is 444 status code and how to avoid it? ](https://scrapfly.io/blog/answers/444-status-code) - [ Q Web scraping - what is HTTP 499 status code? ](https://scrapfly.io/blog/answers/499-status-code) - [ Q Web scraping - what is HTTP 503 status code? ](https://scrapfly.io/blog/answers/503-status-code) ## Ready to scale your web scraping? Anti-bot bypass, browser rendering, and rotating proxies, all in one API. [ Try Scrapfly for FREE ](https://scrapfly.io/register) [ View Documentation ](https://scrapfly.io/docs/onboarding)