Scraper doesn't see the data I see in the browser - why?

When scraping we might notice that some page elements are only visible in the web browser but not in our scraper. This is called dynamic javascript data and it's being created by javascript on page load. If our scraper is not running a full browser to execut javascript it'll never see dynamic elements rendered.

There are many ways to scrape dynamic data like using web browsers:

How to Scrape Dynamic Websites Using Headless Web Browsers

See our introduction tutorial article to scraping using web browsers and automation toolkits like Puppeteer, Selenium and Playwright

How to Scrape Dynamic Websites Using Headless Web Browsers

Alternatively, sometimes dynamic data is already present in the HTML document but in a different location than what we see in the browser. Most commonly the data is hidden in <script> elements as javascript variables and then unpacked into the HTML on page load.

How to Scrape Hidden Web Data

For more see this introduction article which covers how to find hidden web data and popular hidden web data scenarios

How to Scrape Hidden Web Data
Question tagged: Data Parsing, Headless Browsers

Related Posts

How to Scrape Sitemaps to Discover Scraping Targets

Usually to find scrape targets we look at site search or category pages but there's a better way - sitemaps! In this tutorial, we'll be taking a look at how to find and scrape sitemaps for target locations.

Web Scraping Simplified - Scraping Microformats

In this short intro we'll be taking a look at web microformats. What are microformats and how can we take advantage in web scraping? We'll do a quick overview and some examples in Python using extrcut library.

Quick Intro to Parsing JSON with JSONPath in Python

Intro to using Python and JSONPath library and a query language for parsing JSON datasets.