How to click on cookie popups and modal alerts in Selenium?

Modal pop-ups are usually encountered when dealing with cookie consent popups or login request walls when scraping using Selenium. The modal pop-up is created using custom javascript that hides the content on page load and shows some sort of message like this one:

cookie consent popup on web-scraping.dev/login page

There are multiple ways to handle modal pop-ups though these are the two most common ways:

  1. We can click on one of the values like "OK" or "Yes"
  2. We can delete the modal element from the DOM

For example, let's take a look at web-scraping.dev/login page which on page load throws a cookie pop-up:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

driver = webdriver.Chrome()

driver.get("https://web-scraping.dev/login")

# Option #1 - use driver.find_element() to click on the button
try:
    cookie_ok_button = WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.ID, "cookie-ok")))
    cookie_ok_button.click()
except TimeoutException:
    print("no cookie popup")


# Option #2 - delete the popup HTML
#   remove pop up
try:
    cookie_modal = WebDriverWait(driver, 2).until(EC.presence_of_element_located((By.ID, "cookieModal")))
    driver.execute_script("arguments[0].remove();", cookie_modal)
except TimeoutException:
    pass
#   remove grey backgdrop which covers the screen
try:
    modal_backdrop = WebDriverWait(driver, 2).until(EC.presence_of_element_located((By.CLASS_NAME, "modal-backdrop")))
    driver.execute_script("arguments[0].remove();", modal_backdrop)
except TimeoutException:
    pass

driver.quit()

Above, we explore two ways to handle modal pop-ups: clicking a button that would dismiss it and hard removing them from the DOM. Generally, the first approach is more reliable as the real button click can have functionality attached to it like setting a cookie so the pop-up doesn't appear again. For cases when it's a login requirement or advertisement, the second approach is more suited.

Question tagged: Selenium

Related Posts

How to Use Chrome Extensions with Playwright, Puppeteer and Selenium

In this article, we'll explore different useful Chrome extensions for web scraping. We'll also explain how to install Chrome extensions with various headless browser libraries, such as Selenium, Playwright and Puppeteer.

Intro to Web Scraping using Selenium Grid

In this guide, you will learn about installing and configuring Selenium Grid with Docker and how to use it for web scraping at scale.

How to Scrape Google Maps

We'll take a look at to find businesses through Google Maps search system and how to scrape their details using either Selenium, Playwright or ScrapFly's javascript rendering feature - all of that in Python.