How to Scrape With Headless Firefox
Discover how to use headless Firefox with Selenium, Playwright, and Puppeteer for web scraping, including practical examples for each library.
To handle browser dialog pop ups in Playwright like this one:
We need to attach a dialog
handler to the page
object which can be done using page.on("dialog", handler)
method:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
page = browser.new_page()
# create a dialog handler that will check message text and press yes/no
def handle_dialog(dialog):
if "clear your cart" in dialog.message:
print(f'clicking "Yes" to {dialog.message}')
dialog.accept() # press "Yes"
else:
dialog.dismiss() # press "No"
page.on("dialog", handle_dialog)
# to test this, let's try web-scraping.dev cart system:
# add something to cart
page.goto("https://web-scraping.dev/product/1")
page.click(".add-to-cart")
# try clearing cart which raises a dialog that says "are you sure you want to clear your cart?"
page.goto("https://web-scraping.dev/cart")
page.wait_for_selector(".cart-full .cart-item")
page.click(".cart-full .cart-clear")
print(f'items in cart: {page.query_selector(".cart-item .cart-title")}') # should be None
In the example above, attach a dialog handler to our page
object which checks whether the dialog message contains the text "clear your cart" and if so, it clicks "Yes" to clear the cart. Otherwise, it clicks "No" to cancel the dialog.