How to use CSS Selectors in Python?

Python has several popular packages that can parse HTML using CSS selectors.
The most popular one is BeautifulSoup which can execute CSS selectors through the select() and select_one() methods:

from bs4 import BeautifulSoup

soup = BeautifulSoup("""
<a>link 1</a>
<a>link 2</a>
""")

print(soup.select_one('a'))
"<a>link 1</a>"
print(soup.select('a'))
["<a>link 1</a>", "<a>link 2</a>"]

Another popular package is parsel (also used by scrapy) which can execute CSS selectors through the css() method:

from parsel import Selector

soup = Selector("""
<a>link 1</a>
<a>link 2</a>
""")

print(soup.css('a').get())
"<a>link 1</a>"
print(soup.css('a').getall())
["<a>link 1</a>", "<a>link 2</a>"]

Related Posts

Web Scraping With Ruby

Introduction to web scraping with Ruby. How to handle http connections, parse html files for data, best practices, tips and an example project.

Web Scraping With NodeJS and Javascript

In this article we'll take a look at scraping using Javascript through NodeJS. We'll cover common web scraping libraries, frequently encountered challenges and wrap everything up by scraping etsy.com

Parsing HTML with CSS Selectors

Introduction to using CSS selectors to parse web-scraped content. Best practices, available tools and common challenges by interactive examples.