[Answers](https://scrapfly.io/blog)   /  [beautifulsoup](https://scrapfly.io/blog/tag/beautifulsoup)   /  [How to find all links using BeautifulSoup and Python?](https://scrapfly.io/blog/answers/how-to-find-all-links-using-beautifulsoup)   # How to find all links using BeautifulSoup and Python?

 by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) Oct 24, 2022 1 min read [\#beautifulsoup](https://scrapfly.io/blog/tag/beautifulsoup) [\#crawling](https://scrapfly.io/blog/tag/crawling) [\#data-parsing](https://scrapfly.io/blog/tag/data-parsing) 

 [  ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-find-all-links-using-beautifulsoup "Share on LinkedIn")    

 

 

BeautifulSoup is a popular HTML parsing library used in web scraping with Python. With BeautifulSoup, to find all links on the page we can use the `find_all()` method or [CSS selectors](https://scrapfly.io/blog/posts/parsing-html-with-css) and the `select()` method:

python```python
import bs4
soup = bs4.BeautifulSoup("""
<a href="/pricing">Pricing</a>
<a href="https://example.com/blog">Blog</a>
<a href="https://twitter.com/@company">Twitter</a>
""")
links = [node.get('href') for node in soup.find_all("a")]
[
    "/pricing",   
    "https://example.com/blog",
    "https://twitter.com/@company",
]
# or with css selectors:
link = [node.get('href') for node in soup.select('a')]
```



It should be noted that bs4 extracts links as they appear on the page. Links can be:

- Relative to the current website like `/pricing`
- Absolute like like `https://example.com/blog`
- Absolute outbound like `https://twitter.com/@company`

We can convert all relative urls to absolute using `urllib.parse.urljoin` function:

python```python
from urllib.parse import urljoin

base_url = "https://example.com"
links = [urljoin(base_url, link) for link in links]
print(links)
# will print
"https://example.com/pricing"
"https://example.com/blog"
"https://twitter.com/@company"
```



We can also filter out outbound URLs if we want to restrict our scraper to a particular website. For this <https://pypi.org/project/tldextract/> library can be used to find the top level domain (TLD):

python```python
import tldextract

allowed_domain = "example.com"
for link in links:
    tld = tldextract.extract("link").registered_domain
    if tld != allowed_domain:
        continue
    else:
        print(link)
# will print
"https://example.com/pricing"
"https://example.com/blog"
# notice the twitter url is missing
```





 

    



Scale Your Web Scraping

Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits.

  No credit card required  1,000 free API credits  Anti-bot bypass included 

 [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) 

 Not ready? Get our newsletter instead. 

 

## Explore this Article with AI

 [ ChatGPT ](https://chat.openai.com/?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-find-all-links-using-beautifulsoup) [ Gemini ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-find-all-links-using-beautifulsoup) [ Grok ](https://x.com/i/grok?text=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-find-all-links-using-beautifulsoup) [ Perplexity ](https://www.perplexity.ai/search/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-find-all-links-using-beautifulsoup) [ Claude ](https://claude.ai/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-find-all-links-using-beautifulsoup) 



 ## Related Articles

 [  

 python data-parsing 

### How to Parse Web Data with Python and Beautifulsoup

Beautifulsoup is one the most popular libraries in web scraping. In this tutorial, we'll take a hand-on overview of how ...

 

 ](https://scrapfly.io/blog/posts/web-scraping-with-python-beautifulsoup) [  

 python crawling 

### How to Find All URLs on a Domain

Learn how to efficiently find all URLs on a domain using Python and web crawling. Guide on how to crawl entire domain to...

 

 ](https://scrapfly.io/blog/posts/how-to-find-all-urls-on-a-domain) [  

 python scrapeguide 

### How to Scrape YellowPages.com in 2026

Tutorial on how to scrape yellowpages.com business and review data using Python. How to avoid blocking to scrape data at...

 

 ](https://scrapfly.io/blog/posts/how-to-scrape-yellowpages) 

  ## Related Questions

- [ Q How to scrape tables with BeautifulSoup? ](https://scrapfly.io/blog/answers/how-to-scrape-tables-with-beautifulsoup)
- [ Q How to find sibling HTML nodes using BeautifulSoup and Python? ](https://scrapfly.io/blog/answers/how-to-find-siblings-nodes-with-beautifulsoup)
- [ Q How to find HTML element by class with BeautifulSoup? ](https://scrapfly.io/blog/answers/how-to-find-html-elements-by-class-with-beautifulsoup)
- [ Q How to find HTML elements by attribute using BeautifulSoup? ](https://scrapfly.io/blog/answers/how-to-find-html-elements-by-attribute-with-beautifulsoup)
 
  



   



 Extract structured data with AI, **1,000 free credits** [Start Free](https://scrapfly.io/register)