Bernard is Scrapfly's Chief Editor and head of content. He's been a Python developer working closely with web scraping for over 10 years
and creating articles, guides and tutorials for Scrapfly since 2022.
Everything to Know to Start Web Scraping in Python Today
Complete introduction to web scraping using Python: http, parsing, AI, scaling and deployment.
PYTHON
What is HTTP Error 429 Too Many Request and How to Fix it
HTTP 429 is an infamous response code that indicates request throttling or distribution is needed. Let's take a look at how to handle it.
HTTP
BLOCKING
What is Status Code 403 Forbidden and How to Fix it
403 Forbidden HTTP status code mean the client is not allowed to view the resources, but why? Let's take a look at reasons and how to bypass it.
HTTP
How to Use cURL GET Requests
Here's everything you need to know about cURL GET requests and some common pitfalls you should avoid.
CURL
How to Scrape Hidden APIs
In this tutorial we'll be taking a look at scraping hidden APIs which are becoming more and more common in modern dynamic websites - what's the best way to scrape them?
PYTHON
API
Ultimate XPath Cheatsheet for HTML Parsing in Web Scraping
Ultimate companion for HTML parsing using XPath selectors. This cheatsheet contains all syntax explanations with interactive examples.
XPATH
DATA-PARSING
Ultimate CSS Selector Cheatsheet for HTML Parsing
Ultimate companion for HTML parsing using CSS selectors. This cheatsheet contains all syntax explanations with interactive examples.
CSS-SELECTORS
DATA-PARSING
Complete Guide to Web Scraping using Typescript
In this introduction we're taking a look at web scraping using Typescript - increasingly popular typed Javascript language and what scraping challenges it solves.
TYPESCRIPT
INTRO
Crafting Web Scrapers using ChatGPT Code Interpreter is Easy
The new chatgpt code intrepreter feature is an ideal assistant for crafting web scrapers. Here's how it can be used to help with HTML parsing.
PYTHON
AI
How to scrape Local Storage using Headless Browsers
Introduction to scraping local storage - a key value store available in all browsers and used in many modern SPAs - all using headless browsers like playwright.
HEADLESS-BROWSER
How to scrape Threads by Meta using Python (2025 Update)
Guide how to scrape Threads - new social media network by Meta and Instagram - using Python and popular libraries like Playwright and background request capture techniques.
SCRAPEGUIDE
PYTHON
Web Scraping Background Requests with Headless Browsers
In this tutorial we'll be taking a look at a rather new and popular web scraping technique - capturing background requests using headless browsers.
PYTHON
HEADLESS-BROWSER
How to Parse Datetime Strings with Python and Dateparser
Dateparser is a popular Python package for parsing datetime strings. Here's how it can be used in web scraping and how to avoid common problems.
PYTHON
DATA-PARSING
Top 10 Web Scraping Packages for Python
These are the most popular and commonly used 10 Python packages in web scraping. From HTTP connections, browser automation and data validation.
PYTHON
How to Web Scrape with HTTPX and Python
Intro to using Python's httpx library for web scraping. Proxy and user agent rotation and common web scraping challenges, tips and tricks.
HTTPX
PYTHON
Stepping into Footwear Market with Web Scraping
Introduction to data analytics for web scraped data. We'll take a look at how can we take advantage of web scraped data to track luxury footwear market.
FASHION
How to Scrape Goat.com for Fashion Apparel Data in Python
Goat.com is a rising storefront for luxury fashion apparel items. It's known for high quality apparel data so in this tutorial we'll take a look how to scrape it using Python.
SCRAPEGUIDE
FASHION
ECOMMERCE
PYTHON
How to Scrape Fashionphile for Second Hand Fashion Data
In this fashion scrapeguide we'll be taking a look at Fashionphile - another major 2nd hand luxury fashion marketplace. We'll be using Python and hidden web data scraping to grap all of this data in just few lines of code.
SCRAPEGUIDE
PYTHON
ECOMMERCE
FASHION
How to Scrape Sitemaps to Discover Scraping Targets
Usually to find scrape targets we look at site search or category pages but there's a better way - sitemaps! In this tutorial, we'll be taking a look at how to find and scrape sitemaps for target locations.
CRAWLING
PYTHON
NODEJS
DATA-PARSING
How to Scrape Vestiaire Collective for Fashion Product Data
In this fashion scrapeguide we'll be taking a look at Vestiaire Collective - one of the biggest 2nd hand luxury fashion marketplaces. We'll be using hiddden web data scraping to scrape data in just a few lines of Python code.
SCRAPEGUIDE
FASHION
ECOMMERCE
PYTHON
How to Scrape Nordstrom Fashion Product Data
In this guide we'll be taking a look at scraping Nordstrom.com - one of the biggest fashion e-commerce shops. We'll be using hidden web data scraping and Python.
SCRAPEGUIDE
FASHION
ECOMMERCE
PYTHON
How to Scrape StockX e-commerce Data with Python
In this first entry in our fashion data web scraping series we'll be taking a look at StockX.com - a marketplace that treats apparel as stocks and how to scrape it all.
SCRAPEGUIDE
PYTHON
FASHION
ECOMMERCE
How to Bypass Imperva Incapsula when Web Scraping in 2025
In this article we'll take a look at a popular anti bot service Imperva Incapsula anti bot WAF. How does it detect web scrapers and bots and what can we do to prevent our scrapers from being detected?
BLOCKING
How to Bypass Datadome Anti Scraping in 2025
In this article we'll take a look at a popular anti bot service Datadome Anti Bot firewall. How does it detect web scrapers and bots and what can we do to prevent our scrapers from being detected?
BLOCKING
How to Bypass Akamai when Web Scraping in 2025
In this article we'll take a look at a popular anti bot service Akamai Bot Manager. How does it detect web scrapers and bots and what can we do to prevent our scrapers from being detected?
BLOCKING
How to Bypass PerimeterX when Web Scraping in 2025
In this article we'll take a look at a popular anti scraping service PerimeterX. How does it detect web scrapers and bots and what can we do to prevent our scrapers from being detected?
BLOCKING
How to Bypass Cloudflare When Web Scraping in 2025
Cloudflare offers one of the most popular anti scraping service, so in this article we'll take a look how it works and how to bypass it.
BLOCKING
Web Scraping Simplified - Scraping Microformats
In this short intro we'll be taking a look at web microformats. What are microformats and how can we take advantage in web scraping? We'll do a quick overview and some examples in Python using extrcut library.
PYTHON
DATA-PARSING
How to Scrape X.com (Twitter) using Python (2025 Update)
With the news of Twitter dropping free API access we're taking a look at web scraping Twitter using Python for free. In this tutorial we'll cover two methods: using Playwright and Twitter's hidden graphql API.
SCRAPEGUIDE
PYTHON
How to Scrape RightMove Real Estate Property Data
In this scrape guide we'll be taking a look at scraping RightMove.co.uk - one of the most popular real estate listing websites in the United Kingdom. We'll be scraping hidden web data and backend APIs directly using Python.
SCRAPEGUIDE
PYTHON
REAL-ESTATE
How to Scrape Google Search Results in 2025
In this scrape guide we'll be taking a look at how to scrape Google Search - the biggest index of public web. We'll cover dynamic HTML parsing and SERP collection itself.
SCRAPEGUIDE
PYTHON
Introduction to Parsing JSON with Python JSONPath
Intro to using Python and JSONPath library and a query language for parsing JSON datasets.
DATA-PARSING
PYTHON
How to Scrape Ebay Using Python (2025 Update)
In this scrape guide we'll be taking a look at Ebay.com - the biggest peer-to-peer e-commerce portal in the world. We'll be scraping product details and product search.
SCRAPEGUIDE
PYTHON
How to Rate Limit Async Requests in Python
Quick tutorial on how to limit asynchronous python connections when web scraping. This can reduce and balance out web scraping speed to avoid scraping pages too fast and blocking.
PYTHON
BLOCKING
How to Scrape Zoopla Real Estate Property Data in Python
Scrape guide for web scraping Zoopla.com for real estate property data. In this tutorial we'll be using Python and hidden web data sraping as well as reverse engineer search and sitemaps systems.
SCRAPEGUIDE
PYTHON
REAL-ESTATE
Quick Intro to Parsing JSON with JMESPath in Python
Introduction to JMESPath - JSON query language which is used in web scraping to parse JSON datasets for scrape data.
DATA-PARSING
PYTHON
How to Scrape Redfin Real Estate Property Data in Python
Tutorial on how to scrape Redfin.com sale and rent property data, using Python and how to avoid blocking to scrape at scale.
SCRAPEGUIDE
PYTHON
REAL-ESTATE
How to Scrape Real Estate Property Data using Python
Introduction to scraping real estate property data. What is it, why and how to scrape it? We'll also list dozens of popular scraping targets and common challenges.
SCRAPEGUIDE
PYTHON
REAL-ESTATE
How to Scrape Idealista.com
In this scrape guide we'll be taking a look at Idealista.com - biggest real estate website in Spain, Portugal and Italy.
SCRAPEGUIDE
PYTHON
REAL-ESTATE
How to Scrape Realtor.com - Real Estate Property Data
In this scrape guide we'll be taking a look at real estate property scraping from Realtor.com. We'll also build a tracker scraper that checks for new listings or price changes.
SCRAPEGUIDE
PYTHON
REAL-ESTATE
How to Scrape Hidden Web Data
The visible HTML doesn't always represent the whole dataset available on the page. In this article, we'll be taking a look at scraping of hidden web data. What is it and how can we scrape it using Python?
PYTHON
DATA-PARSING
How to Ensure Web Scrapped Data Quality
Ensuring consitent web scrapped data quality can be a difficult and exhausting task. In this article we'll be taking a look at two populat tools in Python - Cerberus and Pydantic - and how can we use them to validate data.
PYTHON
DATA-PARSING
How to Turn Web Scrapers into Data APIs
Delivering web scraped data can be a difficult problem - what if we could scrape data on demand? In this tutorial we'll be building a data API using FastAPI and Python for real time web scraping.
PYTHON
API
PROJECT
How to Scrape Glassdoor (2025 update)
In this web scraping tutorial we'll take a look at Glassdoor - a major resource for company review, job listings and salary data.
PYTHON
SCRAPEGUIDE
Web Scraping with Playwright and Python
Playwright is the new, big browser automation toolkit - can it be used for web scraping? In this introduction article, we'll take a look how can we use Playwright and Python to scrape dynamic websites.
PYTHON
HEADLESS-BROWSER
How to Rotate Proxies in Web Scraping
In this article we explore proxy rotation. How does it affect web scraping success and blocking rates and how can we smartly distribute our traffic through a pool of proxies for the best results.
PYTHON
PROXIES
Web Scraping Speed: Processes, Threads and Async
Scaling web scrapers can be difficult - in this article we'll go over the core principles like subprocesses, threads and asyncio and how all of that can be used to speed up web scrapers dozens to hundreds of times.
PYTHON
SCALING
How to Scrape Indeed.com (2025 Update)
In this web scraping tutorial we'll be taking a look at Indeed.com - a popular job listing website. In just few lines of Python code we'll scrape all job listings in particular niche and area.
PYTHON
SCRAPEGUIDE
How to Scrape Algolia Search
In this web scraping tutorial we'll take a look at a search service used in web development - Algolia search API - and how can we scrape it?
PYTHON
HIDDEN-API
PROJECT
How to Crawl the Web with Python
Introduction to web crawling with Python. What is web crawling? How it differs from web scraping? And a deep dive into code, building our own crawler and an example project crawling Shopify-powered websites.
PYTHON
How to Scrape Zoominfo Company Data (2025 Update)
Practical tutorial on how to web scrape public company and people data from Zoominfo.com using Python and how to avoid being blocked using ScrapFly API.
SCRAPEGUIDE
PYTHON
How to Scrape Google Maps
We'll take a look at to find businesses through Google Maps search system and how to scrape their details using either Selenium, Playwright or ScrapFly's javascript rendering feature - all of that in Python.
SCRAPEGUIDE
PYTHON
SELENIUM
PLAYWRIGHT
How to Scrape Wellfound Company Data and Job Listings
Tutorial for web scraping Wellfound.com (previously angel.co) tech startup company and job directory using Python.
SCRAPEGUIDE
PYTHON
How to Scrape Crunchbase in 2025
Tutorial on how to scrape crunchbase.com business and related data using Python. How to avoid blocking to scrape data at scale and other tips.
SCRAPEGUIDE
PYTHON
How to Scrape YellowPages.com in 2025
Tutorial on how to scrape yellowpages.com business and review data using Python. How to avoid blocking to scrape data at scale and other tips.
SCRAPEGUIDE
PYTHON
How to Scrape Amazon.com Product Data and Reviews
This scrape guide covers the biggest e-commerce platform in US - Amazon.com. We'll take a look how to scrape product data and reviews in Python, as well as some common challenges, tips and tricks.
SCRAPEGUIDE
PYTHON
How to Scrape Zillow Real Estate Property Data in Python
Tutorial on how to scrape Zillow.com sale and rent property data, using Python and how to avoid blocking to scrape at scale.
SCRAPEGUIDE
PYTHON
REAL-ESTATE
How to Scrape TripAdvisor.com (2025 Updated)
In this scrape guide, we'll be scraping TripAdvisor.com. We'll take a look how to find hotels and other places using the search system and how to scrape hotel reviews, pricing details and other TripAdvisor data.
SCRAPEGUIDE
PYTHON
How to Scrape Aliexpress.com (2025 Update)
Tutorial on how to scrape Aliexpress.com product, review and pricing data using Python. How to avoid blocking to scrape at scale and other tips.
SCRAPEGUIDE
PYTHON
Creating Search Engine for any Website using Web Scraping
Guide for creating a search engine for any website using web scraping in Python. How to crawl data, index it and display it via js powered GUI.
DATA-PARSING
CRAWLING
SEARCH-ENGINE
How to Scrape Booking.com (2025 Update)
Tutorial on how to scrape booking.com hotel and pricing data using Python. How to avoid blocking to web scrape data at scale and other tips.
SCRAPEGUIDE
PYTHON
Web Scraping With Node-Unblocker
Tutorial on using Node-Unblocker - a nodejs library - to avoid blocking while web scraping and using it to optimize web scraping stacks.
NODEJS
PROXIES
BLOCKING
How to Scrape Instagram in 2025
Tutorial on how to scrape instagram.com user and post data using pure Python. How to scrape instagram without loging in or being blocked.
SCRAPEGUIDE
PYTHON
How to Scrape Walmart.com Product Data (2025 Update)
Tutorial on how to scrape walmart.com product and review data using Python. How to avoid blocking to web scrape data at scale and other tips.
PYTHON
SCRAPEGUIDE
How to Scrape Yelp.com (2025 update)
Tutorial on how to scrape yelp.com business and review data using Python. How to avoid blocking to web scrape data at scale and other tips.
PYTHON
SCRAPEGUIDE
How Headers Are Used to Block Web Scrapers and How to Fix It
Introduction to web scraping headers - what do they mean, how to configure them in web scrapers and how to avoid being blocked.
HTTP
BLOCKING
How to Avoid Web Scraper IP Blocking?
How IP addresses are used in web scraping blocking. Understanding IP metadata and fingerprinting techniques to avoid web scraper blocks.
HTTP
BLOCKING
PROXIES
How Javascript is Used to Block Web Scrapers? In-Depth Guide
Introduction to how javascript is used to detect web scrapers. What's in javascript fingerprint and how to correctly spoof it for web scraping.
BLOCKING
HEADLESS-BROWSER
How TLS Fingerprint is Used to Block Web Scrapers?
TLS fingeprinting is a popular way to identify web scrapers that not many developers are aware of. What is it and how can we fortify our scrapers to avoid being detected?
BLOCKING
5 Tools to Scrape Without Blocking and How it All Works
Tutorial on how to avoid web scraper blocking. What is javascript and TLS (JA3) fingerprinting and what role request headers play in blocking.
BLOCKING
Web Scraping Graphql with Python
Introduction to web scraping graphql powered websites. How to create graphql queries in python and what are some common challenges.
HTTP
GRAPHQL
PYTHON
Web Scraping with Python
Introduction tutorial to web scraping with Python. How to collect and parse public data. Challenges, best practices and an example project.
PYTHON
DATA-PARSING
HTTP
INTRO
Web Scraping With R Tutorial and Example Project
Introduction to web scraping with R language. How to handle http connections, parse html files, best practices, tips and an example project.
R
INTRO
DATA-PARSING
HTTP
Best Web Scraping Proxy Providers (2025 Update)
Analysis and comparison of some of the most popular proxy providers. What makes a good proxy providers? What features and dangers to look out for?
PROXIES
BLOCKING
Top 4 Mobile Proxy Providers for Web Scraping
Mobile proxies are really useful for avoiding web scraper blocking - so, which mobile proxy providers are the best and how to choose the right one?
PROXIES
BLOCKING
Top 5 Residential Proxy Providers for Web Scraping
Residential proxies are the most popular type of proxies used in web scraping. What makes a good residential proxy and what providers are the best?
PROXIES
BLOCKING
The Complete Guide To Using Proxies For Web Scraping
Introduction to proxy usage in web scraping. What types of proxies are there? How to evaluate proxy providers and avoid common issues.
PROXIES
BLOCKING
INTRO
Web Scraping With Ruby
Introduction to web scraping with Ruby. How to handle http connections, parse html files for data, best practices, tips and an example project.
RUBY
HTTP
DATA-PARSING
CSS-SELECTORS
XPATH
INTRO
Web Scraping With NodeJS and Javascript
In this article we'll take a look at scraping using Javascript through NodeJS. We'll cover common web scraping libraries, frequently encountered challenges and wrap everything up by scraping etsy.com
NODEJS
HTTP
DATA-PARSING
CSS-SELECTORS
INTRO
How to Web Scrape with Puppeteer and NodeJS in 2025
Introduction to using Puppeteer in Nodejs for web scraping dynamic web pages and web apps. Tips and tricks, best practices and example project.
PUPPETEER
FRAMEWORK
NODEJS
HEADLESS-BROWSER
DATA-PARSING
Parsing HTML with CSS Selectors
Introduction to using CSS selectors to parse web-scraped content. Best practices, available tools and common challenges by interactive examples.
DATA-PARSING
CSS-SELECTORS
Parsing HTML with Xpath
Introduction to xpath in the context of web-scraping. How to extract data from HTML documents using xpath, best practices and available tools.
DATA-PARSING
PARSEL
XPATH
PYTHON
Web Scraping With PHP 101
Introduction to web scraping with PHP. How to handle http connections, parse html files for data, best practices, tips and an example project.
PHP
HTTP
DATA-PARSING
XPATH
INTRO
Web Scraping With Scrapy: The Complete Guide in 2025
Tutorial on web scraping with scrapy and Python through a real world example project. Best practices, extension highlights and common challenges.
PYTHON
SCRAPY
FRAMEWORK
XPATH
INTRO
Web Scraping with Selenium and Python
Introduction to web scraping dynamic javascript powered websites and web apps using Selenium browser automation library and Python.
PYTHON
SELENIUM
HEADLESS-BROWSER
How to Parse Web Data with Python and Beautifulsoup
Beautifulsoup is one the most popular libraries in web scraping. In this tutorial, we'll take a hand-on overview of how to use it, what is it good for and explore a real -life web scraping example.
BEAUTIFULSOUP
DATA-PARSING
PYTHON
How to Scrape Dynamic Websites Using Headless Web Browsers
Introduction to using web automation tools such as Puppeteer, Playwright, Selenium and ScrapFly to render dynamic websites for web scraping
HEADLESS-BROWSER
PYTHON
SELENIUM
PUPPETEER
PLAYWRIGHT
