Selenium Knowledgebase

Selenium is one of the most popular tools for web scraping and web automation. It allows developers to control real web browsers like Chrome, Firefox, and Safari programmatically, making it a powerful choice for scraping dynamic websites that rely on JavaScript.

While Selenium is quite old it still holds up in web scraping and web automation tasks, particularly because of big community and ecosystem around it. It supports multiple programming languages, including Python, Java, C#, and JavaScript, making it accessible to a wide range of developers.

Web Scraping with Selenium and Python

Introduction to web scraping dynamic javascript powered websites and web apps using Selenium browser automation library and Python.

See below for more on Selenium in the context of web scraping and data programming 👇

How to handle popup dialogs in Selenium?

To click on a pop-up alert using Selenium the alert_is_present method can be used to wait for and interact with alerts. Here's how.

#selenium

How to click on cookie popups and modal alerts in Selenium?

To click on modal alerts like cookie popups in Selenium we can either find the button and click it or remove the modal elements. Here's how.

#selenium

How to block resources in Selenium and Python?

To block http resources in selenium we need an external proxy. Here's how to setup mitmproxy to block requests and responses in Selenium.

#selenium

How to capture background requests and responses in Selenium?

To capture background requests and response selenium needs to be extended with Selenium-wire. Here's how to do it.

#selenium

How to scroll to the bottom of the page with Selenium?

To scroll to the very bottom of the page the javascript evaluation feature can be used within a while loop. Here's how.

#selenium

Selenium: chromedriver executable needs to be in PATH?

selenium error "chromedriver executable needs to be in PATH" means that chrome driver is not installed or reachable - here's how to fix it.

#selenium

#python

Selenium: geckodriver executable needs to be in PATH?

selenium error "geckodriver executable needs to be in PATH" means that gecko driver is not installed or reachable - here's how to fix it.

#selenium

#python

How to block image loading in Selenium?

To increase Selenium's performance we can block images. To do that with Chrome browser "prefs" launch option can be used. Here's how.

#selenium

#python

Articles Related to Selenium

Bypass Proxy Detection with Browser Fingerprint Impersonation

Stop proxy blocks with browser fingerprint impersonation using this guide for Playwright, Selenium, curl-impersonate & Scrapfly

Guide to SeleniumBase — A Better & Easier Selenium

SeleniumBase streamlines browser automation with simple syntax, cross-browser support, and robust features, perfect for testing and web scraping.

SELENIUM

HEADLESS-BROWSER

Playwright vs Selenium

Explore the key differences between Playwright vs Selenium in terms of performance, web scraping, and automation testing for modern web applications.

HEADLESS-BROWSER

PLAYWRIGHT

SELENIUM

What is a Headless Browser? Top 5 Headless Browser Tools

Quick overview of new emerging tech of browser automation - what exactly are these tools and how are they used in web scraping?

How to Scrape With Headless Firefox

Discover how to use headless Firefox with Selenium, Playwright, and Puppeteer for web scraping, including practical examples for each library.

Selenium Wire Tutorial: Intercept Background Requests

In this guide, we'll explore web scraping with Selenium Wire. We'll define what it is, how to install it, and how to use it to inspect and manipulate background requests.

Web Scraping Dynamic Web Pages With Scrapy Selenium

Learn how to scrape dynamic web pages with Scrapy Selenium. You will also learn how to use Scrapy Selenium for common scraping use cases, such as waiting for elements, clicking buttons and scrolling.

How to use Headless Chrome Extensions for Web Scraping

In this article, we'll explore different useful Chrome extensions for web scraping. We'll also explain how to install Chrome extensions with various headless browser libraries, such as Selenium, Playwright and Puppeteer.

Intro to Web Scraping Using Selenium Grid

In this guide, you will learn about installing and configuring Selenium Grid with Docker and how to use it for web scraping at scale.

SELENIUM

PYTHON

How to Scrape Google Maps

We'll take a look at to find businesses through Google Maps search system and how to scrape their details using either Selenium, Playwright or ScrapFly's javascript rendering feature - all of that in Python.