[Answers](https://scrapfly.io/blog) / [beautifulsoup](https://scrapfly.io/blog/tag/beautifulsoup) / [How to scrape tables with BeautifulSoup?](https://scrapfly.io/blog/answers/how-to-scrape-tables-with-beautifulsoup) # How to scrape tables with BeautifulSoup? by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) Oct 01, 2026 3 min read [\#beautifulsoup](https://scrapfly.io/blog/tag/beautifulsoup) [\#data-parsing](https://scrapfly.io/blog/tag/data-parsing) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-scrape-tables-with-beautifulsoup "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-scrape-tables-with-beautifulsoup&text=How%20to%20scrape%20tables%20with%20BeautifulSoup%3F "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-scrape-tables-with-beautifulsoup "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-scrape-tables-with-beautifulsoup) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-scrape-tables-with-beautifulsoup) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-scrape-tables-with-beautifulsoup) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-scrape-tables-with-beautifulsoup) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fhow-to-scrape-tables-with-beautifulsoup) Use BeautifulSoup when you need control over messy tables, `pandas.read_html` when the markup is clean and you want DataFrames, and a rendered page when the table isn't in the HTML at all. Read headers from `` and select `th` and `td` together, or you'll silently drop row-label columns. ## Which method should you use? All four routes read the same HTML. The choice comes down to how much control you need and whether the table exists in the HTML at all. | Method | Use it when | Cost | |---|---|---| | **BeautifulSoup** | You need control: messy markup, header cells inside the body, picking one table out of several, cleaning values as you go | You write the row loop yourself | | **`pandas.read_html`** | You want every table on the page as DataFrames and the markup is well formed | Needs `lxml` or `html5lib` installed. Values with units or decimal commas stay strings | | **lxml** | You already use XPath, or you parse large documents and want the faster tree | XPath is built in, but CSS selectors need the separate `cssselect` package | | **Browser rendering** | The table isn't in the HTML the server returns, or the page is behind bot protection | An HTTP client alone won't get you the table | Before picking a library, check whether the table is in the HTML. If view-source doesn't contain the rows, no parser will find them, so skip to [browser-rendered tables](#browser-rendered-and-protected-tables). For parser selection beyond tables, see [BeautifulSoup alternatives](https://scrapfly.io/blog/answers/what-are-some-beautifulsoup-alternatives). ## Setup Install every library this page uses: shell ```shell pip install beautifulsoup4 lxml requests pandas ``` `requests` is a third-party HTTP client, and any client works here, including [httpx](https://scrapfly.io/blog/posts/web-scraping-with-python-httpx). The `"lxml"` parser is fast and forgiving, while `"html.parser"` ships with Python and needs no extra install. ## Retrieve Table Data To start, let's look at our target table on [web-scraping.dev/product/1](https://web-scraping.dev/product/1): We'll request the page to retrieve the table data from its HTML: python ```python from bs4 import BeautifulSoup import requests response = requests.get("https://web-scraping.dev/product/1") soup = BeautifulSoup(response.text, "lxml") ``` This fetches the page and builds the soup object that the next section parses. ## Parse HTML Tables The page has two tables with the `table-product` class. The first has no header, and the second has a ``. Instead of picking one by index, which breaks when the page changes, we select with a [CSS](https://scrapfly.io/blog/posts/parsing-html-with-css) selector and keep the table that has a header: python ```python from bs4 import BeautifulSoup import requests response = requests.get("https://web-scraping.dev/product/1") soup = BeautifulSoup(response.text, "lxml") # pick the product table that has a header row table = next(t for t in soup.select("table.table-product") if t.find("thead")) head = table.find("thead") headers = [c.get_text(strip=True) for c in head.find_all(["th", "td"])] if head else [] body = table.find("tbody") or table rows = [] for tr in body.find_all("tr", recursive=False): cells = tr.find_all(["th", "td"], recursive=False) if cells: rows.append([c.get_text(strip=True) for c in cells]) print(headers) for row in rows: print(row) ``` If the table has an `id`, `soup.find("table", {"id": "..."})` is the most stable selector of all. The rest of the code is built around three rules: - Read the header from `` when it exists instead of assuming the first row is the header. Without one, `headers` stays empty and the header row comes through as `rows[0]`. - Select `["th", "td"]` in each row, not `td` alone, so row-label cells survive. - `recursive=False` stops a nested table's rows from being pulled into the outer table. The second rule catches a quiet bug. Stats and spec tables often start body rows with ``. A loop that reads only `td` cells returns `['72', '2207']` under the headers `['Player', 'Games', 'Points']`, which leaves every row one column short without raising an error. Here's the output: text ```text ['Version', 'Package Weight', 'Package Dimension', 'Variants', 'Delivery Type'] ['Pack 1', '1,00 kg', '100x230 cm', '6 available', '1 Day shipping'] ['Pack 2', '2,11 kg', '200x460 cm', '6 available', '1 Day shipping'] ['Pack 3', '3,22 kg', '300x690 cm', '6 available', '1 Day shipping'] ['Pack 4', '4,33 kg', '400x920 cm', '6 available', '1 Day shipping'] ['Pack 5', '5,44 kg', '500x1150 cm', '6 available', '1 Day shipping'] ``` For more details on parsing with BeautifulSoup, refer to our dedicated guide. [How to Parse Web Data with Python and BeautifulsoupBeautifulsoup is one the most popular libraries in web scraping. In this tutorial, we'll take a hand-on overview of how to use it, what is it good for and explore a real -life web scraping example.](https://scrapfly.io/blog/posts/web-scraping-with-python-beautifulsoup) ## Use pandas.read\_html for clean tables When the markup is clean, `pandas.read_html` is the shortest path: python ```python import io import requests import pandas as pd html = requests.get("https://web-scraping.dev/product/1").text tables = pd.read_html(io.StringIO(html)) # every table on the page df = pd.read_html(io.StringIO(html), match="Package Weight")[0] print(df) print(df["Package Weight"].dtype, repr(df["Package Weight"][0])) ``` text ```text Version Package Weight Package Dimension Variants Delivery Type 0 Pack 1 1,00 kg 100x230 cm 6 available 1 Day shipping 1 Pack 2 2,11 kg 200x460 cm 6 available 1 Day shipping 2 Pack 3 3,22 kg 300x690 cm 6 available 1 Day shipping 3 Pack 4 4,33 kg 400x920 cm 6 available 1 Day shipping 4 Pack 5 5,44 kg 500x1150 cm 6 available 1 Day shipping str '1,00 kg' ``` `read_html` returns a list of every table it can parse, so `match=` is safer than guessing an index. It also needs a parser backend: with neither `lxml` nor `html5lib` installed, it raises an `ImportError` telling you to install lxml. It converts clean numbers, but nothing messier. The `Package Weight` column comes back as strings like `'1,00 kg'` because the values carry a unit and a decimal comma, so you still have to convert them before any math. Passing a URL straight to `read_html` also works, but fetching the HTML yourself keeps headers, sessions, and the rendered route below under your control. Scrapfly #### Extract structured data automatically? Scrapfly's Extraction API uses AI to turn any webpage into structured data — no selectors needed. [Try Free →](https://scrapfly.io/register)## Use pandas.read\_html for clean tables When the markup is clean, `pandas.read_html` is the shortest path: python ```python import io import requests import pandas as pd html = requests.get("https://web-scraping.dev/product/1").text tables = pd.read_html(io.StringIO(html)) # every table on the page df = pd.read_html(io.StringIO(html), match="Package Weight")[0] print(df) print(df["Package Weight"].dtype, repr(df["Package Weight"][0])) ``` text ```text Version Package Weight Package Dimension Variants Delivery Type 0 Pack 1 1,00 kg 100x230 cm 6 available 1 Day shipping 1 Pack 2 2,11 kg 200x460 cm 6 available 1 Day shipping 2 Pack 3 3,22 kg 300x690 cm 6 available 1 Day shipping 3 Pack 4 4,33 kg 400x920 cm 6 available 1 Day shipping 4 Pack 5 5,44 kg 500x1150 cm 6 available 1 Day shipping str '1,00 kg' ``` `read_html` returns a list of every table it can parse, so `match=` is safer than guessing an index. It also needs a parser backend: with neither `lxml` nor `html5lib` installed, it raises an `ImportError` telling you to install lxml. It converts clean numbers, but nothing messier. The `Package Weight` column comes back as strings like `'1,00 kg'` because the values carry a unit and a decimal comma, so you still have to convert them before any math. Passing a URL straight to `read_html` also works, but fetching the HTML yourself keeps headers, sessions, and the rendered route below under your control. ## Use lxml for XPath If you already work with XPath or parse large documents, lxml gets the same result: python ```python import requests import lxml.html doc = lxml.html.fromstring(requests.get("https://web-scraping.dev/product/1").text) table = doc.xpath("//table[contains(@class, 'table-product')][thead]")[0] head_cells = table.xpath(".//thead//tr[1]/*[self::th or self::td]") headers = [c.text_content().strip() for c in head_cells] rows = [ [c.text_content().strip() for c in tr.xpath("./*[self::th or self::td]")] for tr in table.xpath(".//tbody/tr") ] ``` This returns all five rows aligned to the five headers. XPath is built into lxml, but `.cssselect()` raises `ImportError: cssselect does not seem to be installed` unless you install `cssselect` separately. Avoid positional shortcuts like `.//tr[position()>1]`. Position is counted per parent, so with a `` present it drops the first body row and returns four rows instead of five. For more, see the [lxml tutorial](https://scrapfly.io/blog/posts/intro-to-parsing-html-xml-python-lxml) and the [XPath guide](https://scrapfly.io/blog/posts/parsing-html-with-xpath). To use XPath while staying on BeautifulSoup, see [XPath selectors in BeautifulSoup](https://scrapfly.io/blog/answers/can-i-use-xpath-selectors-in-beautifulsoup). ## Browser-rendered and protected tables If `requests.get(url).text` doesn't contain the table markup, the table is rendered client side or the request was blocked. Either render the page or call the API the page itself calls. Check the browser's network tab for a JSON endpoint first, since that's often the cheaper fix. To render the page, or to get past bot protection, fetch the HTML with the [Scrapfly Web Scraping API](https://scrapfly.io/products/web-scraping-api) and feed it to the same parser. Install the SDK first with `pip install scrapfly-sdk`: python ```python from bs4 import BeautifulSoup from scrapfly import ScrapflyClient, ScrapeConfig client = ScrapflyClient(key="YOUR_SCRAPFLY_KEY") result = client.scrape(ScrapeConfig( "https://web-scraping.dev/product/1", render_js=True, # run the page's JavaScript before returning HTML asp=True, # get past bot protection )) soup = BeautifulSoup(result.content, "lxml") # the table parsing code from above works unchanged ``` The parsing code doesn't change, only the way you fetch the HTML. If you'd rather send the HTML and get structured JSON back than write a row loop, use the [Extraction API](https://scrapfly.io/products/extraction-api). ## What this doesn't solve - `colspan` and `rowspan`: a spanning cell breaks the assumption that row length equals header length. You have to expand spans yourself, which is where `pandas.read_html` earns its keep. - Nested tables: `find_all("tr")` is recursive by default and pulls inner rows into the outer table unless you pass `recursive=False`. - String values: units, thousands separators, and decimal commas all survive extraction, as the `'1,00 kg'` above shows. - Multi-row headers: the loop above joins every `` cell into one list, so a two-row header comes out longer than the data rows. `pandas.read_html` returns a MultiIndex instead. - Pagination and lazy loading: the table in the HTML may be page one of many. To save the rows you extract, see [how to export an HTML table to a spreadsheet](https://scrapfly.io/blog/answers/html-table-to-xlsx-python-beautifulsoup). [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents - [Which method should you use?](#which-method-should-you-use) - [Setup](#setup) - [Retrieve Table Data](#retrieve-table-data) - [Parse HTML Tables](#parse-html-tables) - [Use pandas.read\_html for clean tables](#use-pandas-read-html-for-clean-tables) - [Use pandas.read\_html for clean tables](#use-pandas-read-html-for-clean-tables) - [Use lxml for XPath](#use-lxml-for-xpath) - [Browser-rendered and protected tables](#browser-rendered-and-protected-tables) - [What this doesn't solve](#what-this-doesn-t-solve) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ python data-parsing ### How to Parse Web Data with Python and Beautifulsoup Beautifulsoup is one the most popular libraries in web scraping. In this tutorial, we'll take a hand-on overview of how ... ](https://scrapfly.io/blog/posts/web-scraping-with-python-beautifulsoup) [ http nodejs ### How to Set Axios Headers: Complete Guide with Examples (2026) Learn how to set, manage, and troubleshoot Axios headers: per-request config, global defaults, instances, interceptors, ... ](https://scrapfly.io/blog/posts/guide-to-javascript-axios-headers) [ http blocking ### How Headers Are Used to Block Web Scrapers and How to Fix It Introduction to web scraping headers - what do they mean, how to configure them in web scrapers and how to avoid being b... ](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-headers) ## Related Questions - [ Q How to find sibling HTML nodes using BeautifulSoup and Python? ](https://scrapfly.io/blog/answers/how-to-find-siblings-nodes-with-beautifulsoup) - [ Q How to select values between two nodes in BeautifulSoup and Python? ](https://scrapfly.io/blog/answers/how-to-select-values-between-two-elements-in-beautifulsoup) - [ Q How to find HTML elements by attribute using BeautifulSoup? ](https://scrapfly.io/blog/answers/how-to-find-html-elements-by-attribute-with-beautifulsoup) - [ Q How to find HTML element by class with BeautifulSoup? ](https://scrapfly.io/blog/answers/how-to-find-html-elements-by-class-with-beautifulsoup) Extract structured data with AI, **1,000 free credits** [Start Free](https://scrapfly.io/register)