How to find sibling HTML nodes using BeautifulSoup and Python?

When web scraping, sometimes it's easier to select a value by finding its sibling first. Using Python and Beautifulsoup, to find element siblings we can use find() and find_all() methods or CSS selectors and the select() method:

import bs4

soup = bs4.BeautifulSoup("""
<span class="price">Price including shipping:</span>
<span>83.13</span>
""")

# using find() method:
soup.find('span', class_="price").find_next_sibling('span').text
"83.13"

# using CSS selectors and `~` notation:
soup.select('.price ~ span')[0].text
# or
soup.select_one('.price ~ span').text
"83.13"

Provided by Scrapfly

This knowledgebase is provided by Scrapfly data APIs, check us out! 👇