How to use XPath selectors in Python?

The most popular package that implements XPath selectors in Python is lxml. We can use the xpath() method to find all matching values:

from lxml import etree

tree = etree.fromstring("""
<div>
    <a>link 1</a>
    <a>link 2</a>
</div>
""")
for result in tree.xpath("//a"):
    print(result.text)
"link 1"
"link 2"

However, in web scraping the recommended way is to use the parsel package. It's based on lxml and providers a more consistent behavior when working with HTML content:

from parsel import Selector

selector = Selector("""
<div>
    <a>link 1</a>
    <a>link 2</a>
</div>
""")

selector.xpath("//a").getall()
['<a>link 1</a>', '<a>link 2</a>']

Related Articles

How to Parse XML

In this article, we'll explain about XML parsing. We'll start by defining XML files, their format and how to navigate them for data extraction.

PYTHON
CSS-SELECTORS
XPATH
DATA-PARSING
How to Parse XML

Parsing HTML with Xpath

Introduction to xpath in the context of web-scraping. How to extract data from HTML documents using xpath, best practices and available tools.

DATA-PARSING
PARSEL
XPATH
PYTHON
Parsing HTML with Xpath

Ultimate Guide to JSON Parsing in Python

Learn JSON parsing in Python with this ultimate guide. Explore basic and advanced techniques using json, and tools like ijson and nested-lookup

DATA-PARSING
PYTHON
Ultimate Guide to JSON Parsing in Python

What is Parsing? From Raw Data to Insights

Learn about the fundamentals of parsing data, across formats like JSON, XML, HTML, and PDFs. Learn how to use Python parsers and AI models for efficient data extraction.

DATA-PARSING
PYTHON
AI
What is Parsing? From Raw Data to Insights

Intro to Parsing HTML and XML with Python and lxml

In this tutorial, we'll take a deep dive into lxml, a powerful Python library that allows for parsing HTML and XML effectively. We'll start by explaining what lxml is, how to install it and using lxml for parsing HTML and XML files. Finally, we'll go over a practical web scraping with lxml.

PYTHON
TOOLS
DATA-PARSING
Intro to Parsing HTML and XML with Python and lxml

Web Scraping to Google Sheets

Google sheets is an easy to store scraped data. In this tutorial we'll take a look at how to use this free online database for storing scraped data!

PYTHON
PROJECT
DATA-PARSING
Web Scraping to Google Sheets