How to Scrape With Headless Firefox
Discover how to use headless Firefox with Selenium, Playwright, and Puppeteer for web scraping, including practical examples for each library.
When web scraping using Puppeteer and Python to capture background requests and responses we can use the page.on()
method to add callbacks on request and response events:
const puppeteer = require('puppeteer');
function run() {
const browser = await puppeteer.launch();
const page = await browser.newPage();
// capture background requests:
await page.setRequestInterception(true);
page.on('request', request => {
if (request.resourceType() === 'xhr') {
console.log(request):
// we can block these requests with:
request.abort();
} else {
request.continue();
}
});
// capture background responses:
page.on('response', response => {
if (response.resourceType() === 'xhr') {
console.log(response);
}
})
await browser.close();
}
run();
Often these background requests can contain important dynamic data. Blocking some requests can also reduce the bandwidth used by the scraper, for more on that see How to block resources in Puppeteer?
This knowledgebase is provided by Scrapfly — a web scraping API that allows you to scrape any website without getting blocked and implements a dozens of other web scraping conveniences. Check us out 👇