What is a Headless Browser? Top 5 Headless Browser Tools
Quick overview of new emerging tech of browser automation - what exactly are these tools and how are they used in web scraping?
In this guide, we'll explore handling popups in Playwright. To start, let's replicate a dialog example:
After following the above steps, you will encounter the below popup window:
First, we'll start by emulating the above popup event within Playwright:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
page = browser.new_page()
# add something to cart
page.goto("https://web-scraping.dev/product/1")
page.click(".add-to-cart")
# clear out the cart to trigger the popup event
page.goto("https://web-scraping.dev/cart")
page.wait_for_selector(".cart-full .cart-item")
page.click("(//button[contains(text(),'Clear')])[2]")
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();
// add something to cart
await page.goto('https://web-scraping.dev/product/1');
await page.click('.add-to-cart');
// clear out the cart to trigger the popup event
await page.goto('https://web-scraping.dev/cart');
await page.waitForSelector('.cart-full .cart-item');
await page.click("(//button[contains(text(),'Clear')])[2]");
await browser.close();
})();
Here, start a new Playwright instance, and create a new page. Then, we use the new tab to request the product web page, add it to the cart, and clear it to trigger the popup dialog.
Next, let's intercept the dialog action to handle it:
from playwright.sync_api import sync_playwright
# create a dialog handler that will check message text and press yes/no
def handle_dialog(dialog):
if "clear your cart" in dialog.message:
print(f'clicking "Yes" to {dialog.message}')
dialog.accept() # press "Yes"
else:
dialog.dismiss() # press "No"
page.on("dialog", handle_dialog)
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
page = browser.new_page()
# intercept popup dialog with the handle_dialog function
page.on("dialog", handle_dialog) # new
# add something to cart
page.goto("https://web-scraping.dev/product/1")
page.click(".add-to-cart")
# clear out the cart to trigger the popup event
page.goto("https://web-scraping.dev/cart")
page.wait_for_selector(".cart-full .cart-item")
page.click("(//button[contains(text(),'Clear')])[2]")
# verify that cart is clear
print(f'items in cart: {page.query_selector(".cart-item .cart-title")}') # should be None
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();
// Create a dialog handler that will check the message text and press Yes/No
page.on('dialog', async dialog => {
if (dialog.message().includes('clear your cart')) {
console.log(`clicking "Yes" to ${dialog.message()}`);
await dialog.accept(); // press "Yes"
} else {
await dialog.dismiss(); // press "No"
}
});
// Add something to cart
await page.goto('https://web-scraping.dev/product/1');
await page.click('.add-to-cart');
// Clear out the cart to trigger the popup event
await page.goto('https://web-scraping.dev/cart');
await page.waitForSelector('.cart-full .cart-item');
await page.click("(//button[contains(text(),'Clear')])[2]");
// Verify that cart is clear
const cartItem = await page.$('.cart-item .cart-title');
console.log(`items in cart: ${cartItem}`); // should be null
await browser.close();
})();
Here, we define a handle_dialog
function to click the pop up. Then, we intercept the dialog and click the popup with the method: page.on("dialog", handle_dialog)
.
We can handle popups in Playwright successfully. For further details on web scraping with Playwright, refer to our dedicated guide.
This knowledgebase is provided by Scrapfly — a web scraping API that allows you to scrape any website without getting blocked and implements a dozens of other web scraping conveniences. Check us out 👇