JSONL vs JSON
Learn the differences between JSON and JSONLines, their use cases, and efficiency. Why JSONLines excels in web scraping and real-time processing
Scraper doesn't see the data I see in the browser - why?
When scraping we might notice that some page elements are only visible in the web browser but not in our scraper. This is called dynamic javascript data and it's being created by javascript on page load. If our scraper is not running a full browser to execut javascript it'll never see dynamic elements rendered.
There are many ways to scrape dynamic data like using web browsers:
How to Scrape Dynamic Websites Using Headless Web Browsers
See our introduction tutorial article to scraping using web browsers and automation toolkits like Puppeteer, Selenium and Playwright
Alternatively, sometimes dynamic data is already present in the HTML document but in a different location than what we see in the browser. Most commonly the data is hidden in <script> elements as javascript variables and then unpacked into the HTML on page load.
How to Scrape Hidden Web Data
For more see this introduction article which covers how to find hidden web data and popular hidden web data scenarios
Provided by Scrapfly
This knowledgebase is provided by Scrapfly data APIs, check us out! 👇
- Web Scraping API - scrape without blocking, control cloud browsers, and more.
- Extraction API - AI and LLM for parsing data.
- Screenshot API - capture pages or elements with no blocks.
Web Scraping and HTML Parsing with Jsoup and Java
Learn how to harness the power of jsoup, a lightweight and efficient Java library for web scraping and HTML parsing.
JSON vs XML: Key Differences and Modern Uses
JSON and XML are two major data formats encountered in web development — here's how they differ and which is one better for your use case.
What is Parsing? From Raw Data to Insights
Learn about the fundamentals of parsing data, across formats like JSON, XML, HTML, and PDFs. Learn how to use Python parsers and AI models for efficient data extraction.