Scraper doesn't see the data I see in the browser - why?

When scraping we might notice that some page elements are only visible in the web browser but not in our scraper. This is called dynamic javascript data and it's being created by javascript on page load. If our scraper is not running a full browser to execut javascript it'll never see dynamic elements rendered.

There are many ways to scrape dynamic data like using web browsers:

How to Scrape Dynamic Websites Using Headless Web Browsers

Introduction to using web automation tools such as Puppeteer, Playwright, Selenium and ScrapFly to render dynamic websites for web scraping

How to Scrape Dynamic Websites Using Headless Web Browsers

Alternatively, sometimes dynamic data is already present in the HTML document but in a different location than what we see in the browser. Most commonly the data is hidden in <script> elements as javascript variables and then unpacked into the HTML on page load.

How to Scrape Hidden Web Data

The visible HTML doesn't always represent the whole dataset available on the page. In this article, we'll be taking a look at scraping of hidden web data. What is it and how can we scrape it using Python?

How to Scrape Hidden Web Data

Related Articles

How to Web Scrape with Puppeteer and NodeJS in 2025

Introduction to using Puppeteer in Nodejs for web scraping dynamic web pages and web apps. Tips and tricks, best practices and example project.

PUPPETEER
FRAMEWORK
NODEJS
HEADLESS-BROWSER
DATA-PARSING
How to Web Scrape with Puppeteer and NodeJS in 2025

Web Scraping with Playwright and JavaScript

Learn about Playwright - a browser automation toolkit for server side Javascript like NodeJS, Deno or Bun.

PLAYWRIGHT
HEADLESS-BROWSER
NODEJS
Web Scraping with Playwright and JavaScript

Ultimate Guide to JSON Parsing in Python

Learn JSON parsing in Python with this ultimate guide. Explore basic and advanced techniques using json, and tools like ijson and nested-lookup

DATA-PARSING
PYTHON
Ultimate Guide to JSON Parsing in Python

Guide to Parsel - the Best HTML Parsing in Python

Learn to extract data from websites with Parsel, a Python library for HTML parsing using CSS selectors and XPath.

DATA-PARSING
PARSEL
Guide to Parsel - the Best HTML Parsing in Python

JSONL vs JSON

Learn the differences between JSON and JSONLines, their use cases, and efficiency. Why JSONLines excels in web scraping and real-time processing

DATA-PARSING
JSONL vs JSON

Guide to SeleniumBase — A Better & Easier Selenium

SeleniumBase streamlines browser automation with simple syntax, cross-browser support, and robust features, perfect for testing and web scraping.

SELENIUM
HEADLESS-BROWSER
Guide to SeleniumBase — A Better & Easier Selenium