Web scraping is mostly connection and data programming so using a web language for scraping seems like a natural fit, so can we scrape using javascript?
In this tutorial, we'll learn web scraping with NodeJS and Javascript. We'll cover an in-depth look at HTTP connections, HTML parsing, popular web scraping libraries and common challenges and web scraping idioms.
Finally, we'll finish everything off with an example web scraping project - https://www.etsy.com/ product scraper that illustrates two major challenges encountered when web scraping in NodeJS: cookie tracking and CSRF tokens.
Overview and Setup
NodeJS in web scraping is mostly known because of Puppeteer browser automation toolkit. Using web browser automation for web scraping has a lot of benefits, though it's a complex and resource-heavy approach to javascript web scraping.
With a little reverse engineering and a few clever nodeJS libraries we can achieve similar results without the entire overhead of a web browser!
In this article, we'll focus on a few tools in particular. For connection, we'll be using axios HTTP client and for parsing we'll focus on cheerio HTML tree parser, let's install them using these command line instructions:
Connection is a vital part of every web scraper and NodeJS has a big ecosystem of HTTP clients, though in this tutorial we'll be using the most popular one - axios.
HTTP in a Nutshell
To collect data from a public resource, we need to establish a connection with it first. Most of the web is served over HTTP. This protocol can be summarized as: client (our scraper) sends a request for a specific document and the server replies with the requested document or an error - a very straight-forward exchange:
illustration of a standard http exchange
As you can see in this illustration: we send a request object which consists of method (aka type), location and headers. In turn, we receive a response object which consists of status code, headers and document content itself.
In our axios example this looks something like this:
Though for node js web scraping we need to know few key details about requests and responses: method types, headers, cookies... Let's take a quick overview.
Request Methods
HTTP requests are conveniently divided into a few types that perform a distinct function. Most commonly in web scraping we use:
GET requests a document - most commonly used method in scraping.
POST sends a document to receive one. For example, this is used in form submissions like login, search etc.
HEAD checks the state of a resources. This is mostly used to check whether a web page has updated it's contents as these type of requests are super fast.
Other methods aren't as commonly encountered but it's good to be aware of them nevertheless:
PATCH requests are intended to update a document.
PUT requests are intended to either create a new document or update it.
DELETE requests are intended to delete a document.
Request Location - The URL
URL (Universal Resource Location) is the most important part of our request - it tells where our nodejs scraper should look for the resources. Though URLs can be quite complicated, let's take a look at how they are structured:
Example of a URL structure
Here, we can visualize each part of a URL:
protocol - is either http or https.
host - is the address/domain of the server.
location - is the location of the resource we are requesting.
parameters - allows customizing of a resource. For example language=en would give us the English version of the resource.
If you're ever unsure of a URL's structure, you can always fire up Node's interactive shell (node in the terminal) and let it figure it out for you:
Request headers indicate meta information about our request. While it might appear like request headers are just minor metadata details in web scraping, they are extremely important.
Headers contain essential details about the request, like who's requesting the data? What type of data they are expecting? Getting these wrong might result in a scraping error.
Let's take a look at some of the most important headers and what they mean:
User-Agent is an identity header that tells the server who's requesting the document.
# example user agent for Chrome browser on Windows operating system:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
Whenever you visit a web page in your web browser identifies itself with a User-Agent string that looks something like "Browser Name, Operating System, Some version numbers".
User-agent helps the server to determine whether to serve or deny the client. When scraping we want to blend in to prevent being blocked so it's best to set user agent to look like that one of a browser.
Cookie is used to store persistent data. This is a vital feature for websites to keep track of user state: user logins, configuration preferences etc.
Accept headers (also Accept-Encoding, Accept-Language etc.) contain information about what sort of content we're expecting. Generally when web-scraping we want to mimic this of one of the popular web browsers, like Chrome browser use:
X- prefixed headers are special custom headers. These are important to keep an eye on when web scraping, as they might configure important functionality of the scraped website/webapp.
These are a few of the most important observations, for more see the extensive full documentation page: MDN HTTP Headers
Response Status Code
Once we send our request we'll eventually receive a response and the first thing we'll notice is the status code. Status codes indicate whether the request succeded, failed or needs something more (like authentication/login).
Let's take a quick look at the status codes that are most relevant to web scraping:
200 range codes generally mean success!
300 range codes tend to mean redirection. In other words, if we request page at /product1.html it might be moved to a new location like /products/1.html.
400 range codes mean the request is malformed or denied. Our node web scraper could be missing some headers, cookies or authentication details.
500 range codes typically mean server issues. The website might be unavailable right now or is purposefully disabling access to our web scraper.
The next thing we notice about our response is the metadata - also known as headers.
When it comes to web scraping, response headers provide some important information for connection functionality and efficiency.
For example, Set-Cookie header requests our client to save some cookies for future requests, which might be vital for website functionality. Other headers such as Etag, Last-Modified are intended to help the client with caching to optimize resource usage.
Finally, just like with request headers, headers prefixed with an X- are custom web functionality headers that we might need to integrate to our scraper.
We took a brief overlook of core HTTP components, and now it's time we give it a go and see how HTTP works in practical Node!
Making GET Requests
Now that we're familiar with the HTTP protocol and how it's used in javascript scraping let's send some requests!
Let's start with a basic GET request:
import axios from 'axios';
const response = await axios.get('https://httpbin.org/get');
console.log(response.data);
Here we're using http://httpbin.org HTTP testing service to retrieve a simple HTML page. When run, this script should print basic details about our made request:
Another document type we can POST is form data type. For this we need to do a bit more work and use form-data package:
import axios from 'axios';
import FormData from 'form-data';
function makeForm(data){
var bodyFormData = new FormData();
for (let key in data){
bodyFormData.append(key, data[key]);
}
return bodyFormData;
}
const resposne = await axios.post('https://httpbin.org/post', makeForm({'query': 'cats', 'page': 1}));
console.log(response.data);
Axios is smart enough to fill in the required header details (like content-type and content-length) based on the data argument. So, if we're sending an object it'll set Content-Type header to application/json and form data to application/x-www-form-urlencoded - pretty convenient!
Setting Headers
As we've covered before our requests must provide some metadata which helps the server to determine what content to return or whether to cooperate with us at all.
Often, this metadata can be used to identify web scrapers and block them, so when scraping we should avoid standing out and mimic a modern web browser.
To start all browsers set User-Agent and Accept headers. To set them in our axios scraper we should create a Client and copy the values from a Chrome web browser:
import axios from 'axios';
const response = await axios.get(
'https://httpbin.org/get',
{headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
}}
);
console.log(response.data);
This will ensure that every request client is making will include these default headers.
When scraping we typically want to apply the same configuration to multiple requests like setting those User-Agent headers for every request our scraper is making to avoid being blocked.
Axios comes with a great shortcut that allows to configure default values for all connections:
Here we created an instance of axios that will apply custom headers, timeout and proxy settings to every request!
Tip: Automatic Cookie Tracking
Sometimes when web-scraping we care about persistent connection state. For websites where we need to login or configure preferences like currency or language - cookies are used to do all of that!
Unfortunately, by default axios doesn't support cookie tracking, however it can be enabled via axios-cookiejar-support extension package:
import axios from 'axios';
import { CookieJar } from 'tough-cookie';
import { wrapper } from 'axios-cookiejar-support';
const jar = new CookieJar();
const session = wrapper(axios.create({ jar }));
async function setLocale(){
// set cookies:
let respSetCookies = await session.get('http://httpbin.org/cookies/set/locale/usa');
// retrieve existing cookies:
let respGetCookies = await session.get('http://httpbin.org/cookies');
console.log(respGetCookies.data);
}
setLocale();
In the example above, we're configuring axios instance with a cookie jar object which allows us to have persistent cookies in our web scraping session. If we run this script we should see:
{ cookies: { locale: 'usa' } }
Now that we're familiar HTTP connection and how can we use it in axios HTTP client package let's take a look at the other half of the web scraping process: parsing HTML data!
Parsing HTML
HTML (HyperText Markup Language) is a text data structure that powers the web. The great thing about it is that it's intended to be machine-readable text content, which is great news for web-scraping as we can easily parse the relevant data with javascript code!
HTML is a tree-type structure that lends easily to parsing. For example, let's take this simple HTML content:
This is a basic HTML document that a simple website might serve. You can already see the tree-like structure just by indentation of the text, but we can even go further and illustrate it:
example of a HTML node tree. Note that branches are ordered (left-to-right)
This tree structure is brilliant for web scraping as we can easily navigate the whole document and extract specific parts that we want.
For example, to find the title of the website, we can see that it's under <body> and under <h1> element. In other words - if we wanted to extract 1000 titles for 1000 different pages, we would write a rule to find body->h1->text rule, but how do we execute this rule?
When it comes to HTML parsing, there are two standard ways to write these rules: CSS selectors and XPATH selectors. Let's see how can we use them in NodeJS and Cheerio next.
CSS Selectors with Cheerio
Cheerio is the most popular HTML parsing package in NodeJS which allows us to use CSS selectors to select specific nodes of an HTML tree.
To use Cheerio we have to create a tree parser object from an HTML string and then we can use a combination of CSS selectors and element functions to extract specific data:
import cheerio from 'cheerio';
const tree = cheerio.load(`
<head>
<title>My Website</title>
</head>
<body>
<div class="content">
<h1>First blog post</h1>
<p>Just started this blog!</p>
<a href="http://scrapfly.io/blog">Checkout My Blog</a>
</div>
</body>
`);
console.log({
// we can extract text of the node:
title: tree('.content h1').text(),
// or a specific attribute value:
url: tree('.content a').attr('href')
});
In the example above, we're loading Cheerio with our example HTML document and highlighting two ways of selecting relevant data. To select the text of an HTML element we're using text() method and to select a specific attribute we're using the attr() method.
XPath Selectors with Xmldom
While CSS selectors are short, robust and easy to read sometimes when dealing with complex web pages we might need something more powerful. For that nodeJS also supports XPATH selectors via libraries like xpath and @xmldom/xmldom:
import xpath from 'xpath';
import { DOMParser } from '@xmldom/xmldom'
const tree = new DOMParser().parseFromString(`
<head>
<title>My Website</title>
</head>
<body>
<div class="content">
<h1>First blog post</h1>
<p>Just started this blog!</p>
<a href="http://scrapfly.io/blog">Checkout My Blog</a>
</div>
</body>
`);
console.log({
// we can extract text of the node, which returns `Text` object:
title: xpath.select('//div[@class="content"]/h1/text()', tree)[0].data,
// or a specific attribute value, which return `Attr` object:
url: xpath.select('//div[@class="content"]/a/@href', tree)[0].value,
});
Here, we're replicating our Cheerio example in xmldom + xpath setup selecting title text and the URL's href attribute.
We looked into two methods of parsing HTML content with NodeJS: using CSS selectors with Cheerio and using Xpath selectors with xmldom + xpath. Generally, it's best to stick with Cheerio as it complies with HTML standard better and CSS selectors are easier to work with.
Let's put everything we've learned by exploring an example project next!
Example Project: etsy.com
We've learned about HTTP connections using axios and HTML parsing using cheerio and now it's time to put everything together and solidify our knowledge.
In this section, we'll write an example scraper for https://www.etsy.com/ which is a user-driven e-commerce website (like Ebay but for crafts). We chose this example to cover two most popular challenges when web scraping with javascript: session cookies and csrf headers.
We'll write a scraper that scrapes the newest products appearing in the vintage product category:
Let's start off by establishing connection with etsy.com and setting our preferred currency/region:
import cheerio from 'cheerio'
import axios from 'axios';
import { wrapper } from 'axios-cookiejar-support';
import { CookieJar } from 'tough-cookie';
const jar = new CookieJar();
const session = wrapper(
axios.create({
jar: jar,
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
})
);
async function setLocale(currency, region){
let _prewalk = await session.get('https://www.etsy.com/');
let tree = cheerio.load(_prewalk.data);
let csrf = tree('meta[name=csrf_nonce]').attr('content');
try{
let resp = await session.post(
'https://www.etsy.com/api/v3/ajax/member/locale-preferences',
{currency:currency, language:"en-US", region: region},
{headers: {'x-csrf-token': csrf}},
);
}catch (error){
console.log(error);
}
}
await setLocale('USD', 'US');
Here, we are creating an axios instance with cookie tracking support. Then we are connecting to Etsy's homepage and looking for csrf token which allows us to interact with Etsy's backend API. Finally, we're sending preference request to this API which returns some tracking cookies that our cookiejar saves automatically for us.
CSRF token is a special security token used in modern web. It essentially tells the webserver that are continuing our communication and not just randomly popping in somewhere in the middle. In etsy example we started communication by requesting homepage, there we found a token which lets us continue our session. For more on CSRF tokens we recommend this stackoverflow thread
From here, every request we make using our axios instance will include these preference cookies - meaning all of our scrape data will be in USD currency.
With site preferences sorted we can continue with our next step - collect newest product urls in the /vintage/ category:
async function findProducts(category){
let resp = await session.get(
`https://www.etsy.com/c/${category}?explicit=1&category_landing_page=1&order=date_desc`
);
let tree = cheerio.load(resp.data);
return tree('a.listing-link').map(
(i, node) => tree(node).attr('href')
).toArray();
}
console.log(await findProducts('vintage'));
Here, we defined our function which given a category name will return urls from first page. Notice, we've added order=date_desc to sort results in descending order by date to pick up only the latest products.
We're left with implementing product scraping itself:
async function scrapeProduct(url){
let resp = await session.get(url);
let tree = cheerio.load(resp.data);
return {
url: url,
title: tree('h1').text().trim(),
description: tree('p[data-product-details-description-text-content]').text().trim(),
price: tree('div[data-buy-box-region=price] p[class*=title]').text().trim(),
store_url: tree('a[aria-label*="more products from store"]').attr('href').split('?')[0],
images: tree('div[data-component=listing-page-image-carousel] img').map(
(i, node) => tree(node).attr('data-src')
).toArray()
};
}
Similarly to earlier all we're doing in this function is retrieving HTML of the product page and extract product details from the HTML content.
Finally, it's time to put everything together as a runner function:
async function scrapeVintage(){
await setLocale('USD', 'US');
let productUrls = await findProducts('vintage');
return Promise.all(productUrls.map(
(url) => scrapeProduct(url)
))
}
console.log(await scrapeVintage());
Here we're combining all of our defined functions into one scraping task which should produce results like:
Using this example javascript web scraper, we've learned about two important sraping concepts: cookies and headers. We configured currency preferences, learned how to deal with csrf tokens and finally, how to scrape and parse product information!
There are many more challenges in web-scraping, so before we wrap this tutorial up let's take a look at some of them.
Avoiding Blocking with ScrapFly
Unfortunately identifying nodeJS based web scrapers is really easy which can lead to web scraper blocking.
Now, we can we can make requests directly through ScrapFly and take advantage it's many features like javascript rendering which uses a real web browser to fully render a web page:
To wrap up this tutorial let's take a look at frequently asked questions about web scraping in JS:
What's the difference between nodejs and puppeteer in web scraping?
Puppeteer is a popular browser automation library for Nodejs. It is frequently used for web scraping. However, we don't always need a web browser to web scrape. In this article, we've learned how can we use Nodejs with a simple HTTP client to scrape web pages. Browsers are very complicated and expensive to run and upkeep so HTTP client web scrapers are much faster and cheaper.
How to scrape concurrently in NodeJS?
Since NodeJS javascript code is naturally asynchronous we can perform concurrent requests to scrape multiple pages by wrapping a list of scrape promises in Promise.all or Promise.allSettled functions. These async await functions take a list of promise objects and executes them in parallel which can speed up web scraping process hundreds of times:
urls = [...]
async function scrape(url){
...
};
let scrape_promises = urls.map((url) => scrape(url));
await Promise.all(scrape_promises);
How to use proxy in NodeJS?
When scraping at scale we might need to use proxies to prevent blocking. Most NodeJS http client libraries implement proxy support through simple arguments. For example in axios library we can set proxy using sessions:
const session = axios.create({
proxy: {
host: 'http://111.22.33.44', //proxy ip address with protocol
port: 80, // proxy port
auth: {username: 'proxy-auth-username', password: 'proxy-auth-password'} // proxy auth if needed
}
}
)
What is the best nodejs web scraping library?
Web Scraping with Cheerio and Nodejs is the most popular way to scrape without using browser automation (Puppeteer) and Axios is the most popular way to make HTTP requests. Though less popular alternatives like xmldom shouldn't be overlooked as they can help with scraping more complex web pages.
How to click buttons, input text do other browser actions in NodeJS?
Since NodeJS engine is not fully browser compliant we cannot automatically click buttons or submit forms. For this something like Puppeteer needs to be used to automate a real web browser. For more see Web Scraping With a Headless Browser: Puppeteer
Summary
In this extensive introduction article we've introduced ourselves with NodeJS web scraping ecosystem. We looked into using axios as our HTTP client to collect multiple pages and using cheerio/@xmldom/xmldom to parse information from this data using CSS/XPATH selectors.
Finally, we wrapped everything up with an example nodejs web scraper project which scrapes vintage product information from https://www.etsy.com/ and looked into ScrapFly's middleware solution which takes care of difficult web scraping challenges such as scaling and blocking!
Usually to find scrape targets we look at site search or category pages but there's a better way - sitemaps! In this tutorial, we'll be taking a look at how to find and scrape sitemaps for target locations.