Headless Browsers Knowledgebase

Headless browsers are web browsers without a graphical user interface (GUI). They are used in web scraping to automate interactions with web pages, allowing developers to extract data from websites that require JavaScript execution or complex user interactions. Using a headless browser you can do anything a real browser can do:

  • Navigate to web pages and retrieve fully rendered HTML content
  • Fill out forms, type text, interact with web widgets
  • Click buttons, links and any other element
  • Load dynamic content generated by JavaScript

For scraping using headless browsers, tools like Puppeteer, Playwright, and Selenium are commonly used. These tools provide APIs to control headless browsers programmatically, enabling developers to navigate web pages, fill out forms, click buttons, and extract data from dynamically loaded content.

What is a Headless Browser? Top 5 Headless Browser Tools

Quick overview of new emerging tech of browser automation - what exactly are these tools and how are they used in web scraping?

What is a Headless Browser? Top 5 Headless Browser Tools

Headless browsers play an increasingly important role in anti-bot bypass as many anti-bot systems require a web browser to bypass due to strong fingerprinting and javascript execution requirements. They can simulate real user behavior, making it harder for anti-bot systems to detect scraping activities.

See below for more on headless browsers in web scraping and browser automation 👇

How to take screenshots in NodeJS?

Learn how to screenshot in Node.js using Playwright & Puppeteer. Includes installation, concepts, and customization tips.

#screenshots
#headless-browser
#puppeteer
#playwright
#nodejs

How to use headless browsers with scrapy?

To use headless browser with scrapy a plugin like scrapy-playwright can be used. Here's how to use it and what are some other alternatives.

#scrapy
#headless-browser

Scraper doesn't see the data I see in the browser - why?

This means that scraper is not rendereding javascript that is changing the page contents. To verify this disable javascript in your browser.

#data-parsing
#headless-browser

How to find elements by CSS selector in Puppeteer?

To find HTML elements using CSS selectors in Puppeteer the $ and $eval methods can be used. Here's how to use them.

#puppeteer
#headless-browser
#data-parsing

How to find elements by XPath in Puppeteer?

To find elements by XPath using Puppeteer the "$x()" method can be used which will execute XPath selection on the current page DOM.

#puppeteer
#headless-browser
#data-parsing
#xpath

How to get page source in Puppeteer?

To retreive page source in Puppteer the page.content() method can be used. Here's how to use it and what are the possible options.

#puppeteer
#python
#headless-browser

How to load local files in Puppeteer?

To load local files in Puppeteer the file:// URL protocol can be used as the URL protocol prefix which will load file from the file path URI

#puppeteer
#headless-browser

How to save and load cookies in Puppeteer?

To save and load cookies in Puppeteer page.setCookies() and page.cookies() methods can be used. Here's how to do it.

#puppeteer
#headless-browser

Articles Related to Headless Browsers

Web Scraping with Playwright and JavaScript

Learn about Playwright - a browser automation toolkit for server side Javascript like NodeJS, Deno or Bun.

PLAYWRIGHT
HEADLESS-BROWSER
NODEJS
Web Scraping with Playwright and JavaScript

Guide to SeleniumBase — A Better & Easier Selenium

SeleniumBase streamlines browser automation with simple syntax, cross-browser support, and robust features, perfect for testing and web scraping.

SELENIUM
HEADLESS-BROWSER
Guide to SeleniumBase — A Better & Easier Selenium

Playwright vs Selenium

Explore the key differences between Playwright vs Selenium in terms of performance, web scraping, and automation testing for modern web applications.

HEADLESS-BROWSER
PLAYWRIGHT
SELENIUM
Playwright vs Selenium

What is a Headless Browser? Top 5 Headless Browser Tools

Quick overview of new emerging tech of browser automation - what exactly are these tools and how are they used in web scraping?

HEADLESS-BROWSER
PLAYWRIGHT
SELENIUM
PUPPETEER
What is a Headless Browser? Top 5 Headless Browser Tools

How To Take Screenshots In Python?

Learn how to take Python screenshots through Selenium and Playwright, including common browser tips and tricks for customizing web page captures.

SCREENSHOTS
PYTHON
HEADLESS-BROWSER
How To Take Screenshots In Python?

Web Scraping With Cloud Browsers

Introduction cloud browsers and their benefits and a step-by-step setup with self-hosted Selenium-grid cloud browsers.

HEADLESS-BROWSER
SCALING
Web Scraping With Cloud Browsers

How to Scrape Forms

Learn how to scrape forms through a step-by-step guide using HTTP clients and headless browsers.

HEADLESS-BROWSER
PYTHON
HTTPX
INTRO
NODEJS
How to Scrape Forms

How to Scrape With Headless Firefox

Discover how to use headless Firefox with Selenium, Playwright, and Puppeteer for web scraping, including practical examples for each library.

HEADLESS-BROWSER
PUPPETEER
SELENIUM
NODEJS
PLAYWRIGHT
PYTHON
How to Scrape With Headless Firefox

Selenium Wire Tutorial: Intercept Background Requests

In this guide, we'll explore web scraping with Selenium Wire. We'll define what it is, how to install it, and how to use it to inspect and manipulate background requests.

PYTHON
HEADLESS-BROWSER
SELENIUM
TOOLS
Selenium Wire Tutorial: Intercept Background Requests

Web Scraping Dynamic Websites With Scrapy Playwright

Learn about Selenium Playwright. A Scrapy integration that allows web scraping dynamic web pages with Scrapy. We'll explain web scraping with Scrapy Playwright through an example project and how to use it for common scraping use cases, such as clicking elements, scrolling and waiting for elements.

PYTHON
PLAYWRIGHT
SCRAPY
HEADLESS-BROWSER
Web Scraping Dynamic Websites With Scrapy Playwright

Web Scraping Dynamic Web Pages With Scrapy Selenium

Learn how to scrape dynamic web pages with Scrapy Selenium. You will also learn how to use Scrapy Selenium for common scraping use cases, such as waiting for elements, clicking buttons and scrolling.

PYTHON
SCRAPY
HEADLESS-BROWSER
SELENIUM
Web Scraping Dynamic Web Pages With Scrapy Selenium

Scrapy Splash Guide: Scrape Dynamic Websites With Scrapy

Learn about web scraping with Scrapy Splash, which lets Scrapy scrape dynamic web pages. We'll define Splash, cover installation and navigation, and provide a step-by-step guide for using Scrapy Splash.

PYTHON
HEADLESS-BROWSER
FRAMEWORK
SCRAPY
Scrapy Splash Guide: Scrape Dynamic Websites With Scrapy

How to Scrape in Another Language, Currency or Location

Localization allows for adapting websites content by changing language and currency. So, how do we scrape it? We'll take a look at the most common methods for changing language, currency and other locality details in web scraping.

PYTHON
HEADLESS-BROWSER
HTTP
How to Scrape in Another Language, Currency or Location

How to scrape Local Storage using Headless Browsers

Introduction to scraping local storage - a key value store available in all browsers and used in many modern SPAs - all using headless browsers like playwright.

HEADLESS-BROWSER
How to scrape Local Storage using Headless Browsers

Web Scraping Background Requests with Headless Browsers

In this tutorial we'll be taking a look at a rather new and popular web scraping technique - capturing background requests using headless browsers.

PYTHON
HEADLESS-BROWSER
Web Scraping Background Requests with Headless Browsers

Web Scraping with Playwright and Python

Playwright is the new, big browser automation toolkit - can it be used for web scraping? In this introduction article, we'll take a look how can we use Playwright and Python to scrape dynamic websites.

PYTHON
HEADLESS-BROWSER
Web Scraping with Playwright and Python

How Javascript is Used to Block Web Scrapers? In-Depth Guide

Introduction to how javascript is used to detect web scrapers. What's in javascript fingerprint and how to correctly spoof it for web scraping.

BLOCKING
HEADLESS-BROWSER
How Javascript is Used to Block Web Scrapers? In-Depth Guide

How to Web Scrape with Puppeteer and NodeJS in 2025

Introduction to using Puppeteer in Nodejs for web scraping dynamic web pages and web apps. Tips and tricks, best practices and example project.

PUPPETEER
FRAMEWORK
NODEJS
HEADLESS-BROWSER
DATA-PARSING
How to Web Scrape with Puppeteer and NodeJS in 2025

Web Scraping with Selenium and Python

Introduction to web scraping dynamic javascript powered websites and web apps using Selenium browser automation library and Python.

PYTHON
SELENIUM
HEADLESS-BROWSER
Web Scraping with Selenium and Python

How to Scrape Dynamic Websites Using Headless Web Browsers

Introduction to using web automation tools such as Puppeteer, Playwright, Selenium and ScrapFly to render dynamic websites for web scraping

HEADLESS-BROWSER
PYTHON
SELENIUM
PUPPETEER
PLAYWRIGHT
How to Scrape Dynamic Websites Using Headless Web Browsers