How to parse dynamic CSS classes when web scraping?

by scrapecrow Nov 03, 2022

Dynamic class names are becoming increasingly common in modern web landscape which can be a tough challenge in web scraping. Let's take a look at this dynamic class example and how can we parse it:

<div class="pdd fg-black">
    <h2>Product Details</h2>
    <div class="fqv b1">
        <div class="fz g1">Price</div>
        <div class="g2 cvx">22.55</div>
    </div>
</div>

Usually, we'd see some human-like class names that we can rely on using CSS Selectors, however in this case the class names look non-sensical which means these classes are most likely dynamic. Dynamic classes can change at any moment which would break our scraper.

The best way to deal with this issue is to use text-based XPath parsing. In our example above to select the price we can find HTML elements by text and relative relationship. See this interactive example:

Product Details

Price
22.55

In this example, we select an element that has the text of Price and then select the first following sibling for the price value. With this approach even if the class names will change our parser will continue to extract data successfully!

For more on text-based parsing see:

Related Articles

Ultimate Guide to JSON Parsing in Python

Learn JSON parsing in Python with this ultimate guide. Explore basic and advanced techniques using json, and tools like ijson and nested-lookup

DATA-PARSING
PYTHON
Ultimate Guide to JSON Parsing in Python

Guide to Parsel - the Best HTML Parsing in Python

Learn to extract data from websites with Parsel, a Python library for HTML parsing using CSS selectors and XPath.

DATA-PARSING
PARSEL
Guide to Parsel - the Best HTML Parsing in Python

JSONL vs JSON

Learn the differences between JSON and JSONLines, their use cases, and efficiency. Why JSONLines excels in web scraping and real-time processing

DATA-PARSING
JSONL vs JSON

Web Scraping and HTML Parsing with Jsoup and Java

Learn how to harness the power of jsoup, a lightweight and efficient Java library for web scraping and HTML parsing.

DATA-PARSING
JAVA
Web Scraping and HTML Parsing with Jsoup and Java

JSON vs XML: Key Differences and Modern Uses

JSON and XML are two major data formats encountered in web development — here's how they differ and which is one better for your use case.

DATA-PARSING
JSON vs XML: Key Differences and Modern Uses

What is Parsing? From Raw Data to Insights

Learn about the fundamentals of parsing data, across formats like JSON, XML, HTML, and PDFs. Learn how to use Python parsers and AI models for efficient data extraction.

DATA-PARSING
PYTHON
AI
What is Parsing? From Raw Data to Insights