How to use XPath selectors in Python?

The most popular package that implements XPath selectors in Python is lxml. We can use the xpath() method to find all matching values:

from lxml import etree

tree = etree.fromstring("""
<div>
    <a>link 1</a>
    <a>link 2</a>
</div>
""")
for result in tree.xpath("//a"):
    print(result.text)
"link 1"
"link 2"

However, in web scraping the recommended way is to use the parsel package. It's based on lxml and providers a more consistent behavior when working with HTML content:

from parsel import Selector

selector = Selector("""
<div>
    <a>link 1</a>
    <a>link 2</a>
</div>
""")

selector.xpath("//a").getall()
['<a>link 1</a>', '<a>link 2</a>']