How to scroll to the bottom of the page with Selenium?

by scrapecrow Jun 30, 2023

Web scraping modern websites faces numerous challenges, one of them being infinite scroll pages. A common solution for such cases is using Selenium scroll to bottom of the page.

To scroll down a web page with Selenium scripts, we can use the popular window.scrollTo(x, y) JavaScript method. This simple scroll operation navigates to Selenium driver to the specified coordinates.

Since infinite scroll pages require executing multiple vertical scroll operations. In other words, to scroll till end means that there are no more changes recorded in the window height. Let's have a look at an example by scraping web-scraping.dev/testimonials:

from selenium  webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

# initiate a new Chrome webdriver
driver = webdriver.Chrome()

# navigate to the target web page
driver.get("https://web-scraping.dev/testimonials/")

prev_height = -1 
max_scrolls = 100
scroll_count = 0

while scroll_count < max_scrolls:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(1)  # give some time for new results to load
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == prev_height:
        break
    prev_height = new_height
    scroll_count += 1

# Collect all loaded data
elements = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "testimonial")))

results = []
for element in elements:
    text = element.find_element(By.CLASS_NAME, "text").get_attribute('innerHTML')
    results.append(text)

print(f"scraped: {len(results)} results!")

driver.quit()

In the above code snippet, we infinite scroll the page in Selenium webdriver using two parameters: previous and current height. It starts by executing the window.scrollTo method to scroll till the page end, record the new height, and compare the previous and current ones till the maximum height is reached.

For further details into using Selenium for web scraping refer to our dedicated guide.

Web Scraping with Selenium and Python

Introduction to web scraping dynamic javascript powered websites and web apps using Selenium browser automation library and Python.

Web Scraping with Selenium and Python

Related Articles

Bypass Proxy Detection with Browser Fingerprint Impersonation

Stop proxy blocks with browser fingerprint impersonation using this guide for Playwright, Selenium, curl-impersonate & Scrapfly

PROXIES
SELENIUM
PLAYWRIGHT
PUPPETEER
BLOCKING
Bypass Proxy Detection with Browser Fingerprint Impersonation

Guide to SeleniumBase — A Better & Easier Selenium

SeleniumBase streamlines browser automation with simple syntax, cross-browser support, and robust features, perfect for testing and web scraping.

SELENIUM
HEADLESS-BROWSER
Guide to SeleniumBase — A Better & Easier Selenium

Playwright vs Selenium

Explore the key differences between Playwright vs Selenium in terms of performance, web scraping, and automation testing for modern web applications.

HEADLESS-BROWSER
PLAYWRIGHT
SELENIUM
Playwright vs Selenium

What is a Headless Browser? Top 5 Headless Browser Tools

Quick overview of new emerging tech of browser automation - what exactly are these tools and how are they used in web scraping?

HEADLESS-BROWSER
PLAYWRIGHT
SELENIUM
PUPPETEER
What is a Headless Browser? Top 5 Headless Browser Tools

How to Scrape With Headless Firefox

Discover how to use headless Firefox with Selenium, Playwright, and Puppeteer for web scraping, including practical examples for each library.

HEADLESS-BROWSER
PUPPETEER
SELENIUM
NODEJS
PLAYWRIGHT
PYTHON
How to Scrape With Headless Firefox

Selenium Wire Tutorial: Intercept Background Requests

In this guide, we'll explore web scraping with Selenium Wire. We'll define what it is, how to install it, and how to use it to inspect and manipulate background requests.

PYTHON
HEADLESS-BROWSER
SELENIUM
TOOLS
Selenium Wire Tutorial: Intercept Background Requests