JSONL vs JSON
Learn the differences between JSON and JSONLines, their use cases, and efficiency. Why JSONLines excels in web scraping and real-time processing
How to turn HTML to text in Python?
When web scraping, we might need to represent scrape HTML data as plain text. For this we can use BeautifulSoup's get_text() method which extracts all visible HTML text and most importantly ignores invisible details such as <script> elements:
from bs4 import BeautifulSoup
soup = BeautifulSoup("""
<body>
<article>
<h1>Article title</h1>
<p>first paragraph and a <a>link</a></p>
<script>var invisible="javascript variable";</script>
</article>
</body>
""")
# if possible it's best to restrict html to a specific element
element = soup.find('article')
text = element.get_text()
print(text)
"""
Article title
first paragraph and a link
"""
Provided by Scrapfly
This knowledgebase is provided by Scrapfly data APIs, check us out! 👇
- Web Scraping API - scrape without blocking, control cloud browsers, and more.
- Extraction API - AI and LLM for parsing data.
- Screenshot API - capture pages or elements with no blocks.
Web Scraping and HTML Parsing with Jsoup and Java
Learn how to harness the power of jsoup, a lightweight and efficient Java library for web scraping and HTML parsing.
JSON vs XML: Key Differences and Modern Uses
JSON and XML are two major data formats encountered in web development — here's how they differ and which is one better for your use case.
What is Parsing? From Raw Data to Insights
Learn about the fundamentals of parsing data, across formats like JSON, XML, HTML, and PDFs. Learn how to use Python parsers and AI models for efficient data extraction.