How to parse dynamic CSS classes when web scraping?

Dynamic class names are becoming increasingly common in modern web landscape which can be a tough challenge in web scraping. Let's take a look at this dynamic class example and how can we parse it:

<div class="pdd fg-black">
    <h2>Product Details</h2>
    <div class="fqv b1">
        <div class="fz g1">Price</div>
        <div class="g2 cvx">22.55</div>
    </div>
</div>

Usually, we'd see some human-like class names that we can rely on using CSS Selectors, however in this case the class names look non-sensical which means these classes are most likely dynamic. Dynamic classes can change at any moment which would break our scraper.

The best way to deal with this issue is to use text-based XPath parsing. In our example above to select the price we can find HTML elements by text and relative relationship. See this interactive example:

Product Details

Price
22.55

In this example, we select an element that has the text of Price and then select the first following sibling for the price value. With this approach even if the class names will change our parser will continue to extract data successfully!

For more on text-based parsing see:

Related Posts

Quick Intro to Parsing JSON with JMESPath in Python

Introduction to JMESPath - JSON query language which is used in web scraping to parse JSON datasets for scrape data.

How to Scrape Hidden Web Data

The visible HTML doesn't always represent the whole dataset available on the page. In this article, we'll be taking a look at scraping of hidden web data. What is it and how can we scrape it using Python?

How to Ensure Web Scrapped Data Quality

Ensuring consitent web scrapped data quality can be a difficult and exhausting task. In this article we'll be taking a look at two populat tools in Python - Cerberus and Pydantic - and how can we use them to validate data.