When scraping dynamic web pages with Playwright and Python we need to wait for the page to fully load before we retrieve the page source for HTML parsing. Let's explore multiple load event methods to ensure a full web page load!
Selectors
In order to make Playwright wait for page to load, we can use Playwright's wait_for_selector method. It ensures a full page load state waiting for specific elements to appear on the web page:
with sync_playwright() as pw:
browser = pw.chromium.launch(headless=False)
context = browser.new_context(viewport={"width": 1920, "height": 1080})
page = context.new_page()
# go to url
page.goto("https://web-scraping.dev/products")
# wait for element to appear on the page:
page.wait_for_selector("div.products")
# get HTML
print(page.content())
Above, we start by creating a new browser context, navigate to the target web page, and wait for the CSS selector div.products to be visible. Finally, we return the page object once it's fully loaded.
Fixed Timeouts
The seconds waiting process is the wait_for_timeout method. Unlike the previous method, this approach doesn't utilize locating elements on the document. Instead, it instructs the browser to wait for a fixed time:
page.goto("https://web-scraping.dev/products")
page.wait_for_timeout(5000)
Here, we use the wait_for_timeout to explicitly wait for 5 seconds before executing the remaining script actions.
Rendering State
The latest load event we'll use is the wait_for_load_state, it relies on different states of network connections:
- domcontentloaded: Waits for the initial DOM structure to be present without waiting for static files to finish loading.
- networkidle: Waits for all background network operations to be finished loading and is idle for at least 500 milliseconds.
- load: Waits the HTML document and its static files to be fully loaded.
Here's how to use the wait_for_load_state to let Playwright wait for page to load through different states:
page.goto("https://web-scraping.dev/products")
PageMethod("wait_for_load_state", "domcontentloaded"),
PageMethod("wait_for_load_state", "networkidle"),
PageMethod("wait_for_load_state", "load"),
For further details on web scraping with Playwright, refer to our dedicated guide.
Web Scraping with Playwright and Python
Playwright is the new, big browser automation toolkit - can it be used for web scraping? In this introduction article, we'll take a look how can we use Playwright and Python to scrape dynamic websites.
