Web Scraping With a Headless Browser: Puppeteer
Introduction to using Puppeteer in Nodejs for web scraping dynamic web pages and web apps. Tips and tricks, best practices and example project.
To test our Puppeteer web scrapers we might want o use local files instead of public websites. Just like real web browsers Puppeteer can load local files using the file://
URL protocol:
const puppeteer = require('puppeteer');
const path = require('path');
async function run() {
// usual browser startup:
const browser = await puppeteer.launch();
const page = await browser.newPage();
// we can use absolute paths like
await page.goto("file://home/user/projects/test.html"); // linux
await page.goto("file://C:/Users/projects/test.html"); // windows
// or we can use relative paths:
// below will select test.html that is in the same directory as the script
await page.goto(`file:${path.join(__dirname, 'test.html')}`);
console.log(await page.content());
browser.close();
}
run();