Discover how to use headless Firefox with Selenium, Playwright, and Puppeteer for web scraping, including practical examples for each library.
To click on modal popups like the infamous cookie conset alert we can either find and click the agree button or remove it entirely. Here's how.
To click on a pop up dialog or an alert in Playwright we can use dialog event capture using `page.on()` method. Here's how.
To scrape to the bottom of the page we can use javascript evaluation for scrolling in a while loop. Here's how to do it.
To check whether an HTML element is present on the page using Playwright the page.locator() method can be used. Here's how.
since playwright and jupyter are both using asyncio to run playwright in a notebook we must used the async client.
To take page screenshots in playwright we can use page.screenshot() method. Here's how to select areas and how to screenshot them in playwright.
To download files using Playwright we can either simulate the button click or extract the url and download it using HTTP. Here's how.
To persist playwright connection session between program runs we can save and load cookies to/from disk. Here's how.
To load local files as page URLs in Playwright we can use the file:// protocol. Here's how to do it.
To execute XPath selectors in playwright the page.locator() method can be used. Here's how.
To execute CSS selectors on current HTML data in Playwright the page.locator() method can be used. Here's how.
To wait for all content to load in playwright we can use several different options but page.wait_for_selector() is the most reliable one. Here's how to use it.
To capture background requests and response in Playwright we can use request/response interception feature through page.on() method. Here's how.
Blocking non-vital resources can drastically speed up Playwright. To do that page interception feature can be used. Here's how.
Discover how to use headless Firefox with Selenium, Playwright, and Puppeteer for web scraping, including practical examples for each library.
Learn about Selenium Playwright. A Scrapy integration that allows web scraping dynamic web pages with Scrapy. We'll explain web scraping with Scrapy Playwright through an example project and how to use it for common scraping use cases, such as clicking elements, scrolling and waiting for elements.
In this article, we'll explore different useful Chrome extensions for web scraping. We'll also explain how to install Chrome extensions with various headless browser libraries, such as Selenium, Playwright and Puppeteer.
We'll take a look at to find businesses through Google Maps search system and how to scrape their details using either Selenium, Playwright or ScrapFly's javascript rendering feature - all of that in Python.
Introduction to using web automation tools such as Puppeteer, Playwright, Selenium and ScrapFly to render dynamic websites for web scraping