Web scraping is mostly connection and data programming so using a web language for scraping seems like a natural fit, so can we scrape using javascript?
In this tutorial, we'll learn web scraping with NodeJS and Javascript. We'll cover an in-depth look at HTTP connections, HTML parsing, popular web scraping libraries and common challenges and web scraping idioms.
Finally, we'll finish everything off with an example web scraping project - https://www.etsy.com/ product scraper that illustrates two major challenges encountered when web scraping in NodeJS: cookie tracking and CSRF tokens.
NodeJS in web scraping is mostly known because of Puppeteer browser automation toolkit. Using web browser automation for web scraping has a lot of benefits, though it's a complex and resource-heavy approach to javascript web scraping.
With a little reverse engineering and a few clever nodeJS libraries we can achieve similar results without the entire overhead of a web browser!
For more on using browser automation with Puppeteer we have an entire introduction article that covers basic usage, best practices, tips and tricks and an example project!
In this article, we'll focus on a few tools in particular. For connection, we'll be using axios HTTP client and for parsing we'll focus on cheerio HTML tree parser, let's install them using these command line instructions:
$ mkdir scrapfly-etsy-scraper
$ cd scrapfly-etsy-scraper
$ npm install cheerio axios
Connection is a vital part of every web scraper and NodeJS has a big ecosystem of HTTP clients, though in this tutorial we'll be using the most popular one - axios.
To collect data from a public resource, we need to establish a connection with it first. Most of the web is served over HTTP. This protocol can be summarized as: client (our scraper) sends a request for a specific document and the server replies with the requested document or an error - a very straight-forward exchange:
As you can see in this illustration: we send a request object which consists of method (aka type), location and headers. In turn, we receive a response object which consists of status code, headers and document content itself.
In our axios example this looks something like this:
import axios from 'axios';
// send request
response = await axios.get('https://httpbin.org/get');
// print response
console.log(response.data);
Though for node js web scraping we need to know few key details about requests and responses: method types, headers, cookies... Let's take a quick overview.
HTTP requests are conveniently divided into a few types that perform a distinct function. Most commonly in web scraping we use:
GET requests a document - most commonly used method in scraping.POST sends a document to receive one. For example, this is used in form submissions like login, search etc.HEAD checks the state of a resources. This is mostly used to check whether a web page has updated it's contents as these type of requests are super fast.Other methods aren't as commonly encountered but it's good to be aware of them nevertheless:
PATCH requests are intended to update a document.PUT requests are intended to either create a new document or update it.DELETE requests are intended to delete a document.URL (Universal Resource Location) is the most important part of our request - it tells where our nodejs scraper should look for the resources. Though URLs can be quite complicated, let's take a look at how they are structured:
Here, we can visualize each part of a URL:
http or https.language=en would give us the English version of the resource.If you're ever unsure of a URL's structure, you can always fire up Node's interactive shell (node in the terminal) and let it figure it out for you:
$ node
> new URL("http://www.domain.com/path/to/resource?arg1=true&arg2=false")
URL {
href: 'http://www.domain.com/path/to/resource?arg1=true&arg2=false',
origin: 'http://www.domain.com',
protocol: 'http:',
username: '',
password: '',
host: 'www.domain.com',
hostname: 'www.domain.com',
port: '',
pathname: '/path/to/resource',
search: '?arg1=true&arg2=false',
searchParams: URLSearchParams { 'arg1' => 'true', 'arg2' => 'false' },
hash: ''
}
Request headers indicate meta information about our request. While it might appear like request headers are just minor metadata details in web scraping, they are extremely important.
Headers contain essential details about the request, like who's requesting the data? What type of data they are expecting? Getting these wrong might result in a scraping error.
Let's take a look at some of the most important headers and what they mean:
User-Agent is an identity header that tells the server who's requesting the document.
# example user agent for Chrome browser on Windows operating system:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
Whenever you visit a web page in your web browser identifies itself with a User-Agent string that looks something like "Browser Name, Operating System, Some version numbers".
User-agent helps the server to determine whether to serve or deny the client. When scraping we want to blend in to prevent being blocked so it's best to set user agent to look like that one of a browser.
There are many online databases that contain latest user-agent strings of various platforms, like this Chrome user agent list bye whatismybrowser.com
Cookie is used to store persistent data. This is a vital feature for websites to keep track of user state: user logins, configuration preferences etc.
Accept headers (also Accept-Encoding, Accept-Language etc.) contain information about what sort of content we're expecting. Generally when web-scraping we want to mimic this of one of the popular web browsers, like Chrome browser use:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
For more, see list of default Accept header values
X- prefixed headers are special custom headers. These are important to keep an eye on when web scraping, as they might configure important functionality of the scraped website/webapp.
These are a few of the most important observations, for more see the extensive full documentation page: MDN HTTP Headers
Once we send our request we'll eventually receive a response and the first thing we'll notice is the status code. Status codes indicate whether the request succeded, failed or needs something more (like authentication/login).
Let's take a quick look at the status codes that are most relevant to web scraping:
/product1.html it might be moved to a new location like /products/1.html.For more on http status codes, see documentation: HTTP Status definitions by MDN
The next thing we notice about our response is the metadata - also known as headers.
When it comes to web scraping, response headers provide some important information for connection functionality and efficiency.
For example, Set-Cookie header requests our client to save some cookies for future requests, which might be vital for website functionality. Other headers such as Etag, Last-Modified are intended to help the client with caching to optimize resource usage.
For the entire list of all HTTP headers, see MDN HTTP Headers
Finally, just like with request headers, headers prefixed with an X- are custom web functionality headers that we might need to integrate to our scraper.
We took a brief overlook of core HTTP components, and now it's time we give it a go and see how HTTP works in practical Node!
Now that we're familiar with the HTTP protocol and how it's used in javascript scraping let's send some requests!
Let's start with a basic GET request:
import axios from 'axios';
const response = await axios.get('https://httpbin.org/get');
console.log(response.data);
Here we're using http://httpbin.org HTTP testing service to retrieve a simple HTML page. When run, this script should print basic details about our made request:
{
args: {},
headers: {
Accept: 'application/json, text/plain, */*',
Host: 'httpbin.org',
'User-Agent': 'axios/0.25.0',
},
origin: '180.111.222.223',
url: 'https://httpbin.org/get'
}
POST type requests are used to interact with the website through its interactive features like login, search functionality or result filtering.
For these requests our scraper needs to send something to receive the response. That something is usually a JSON document:
import axios from 'axios';
const response = await axios.post('https://httpbin.org/post', {'query': 'cats', 'page': 1});
console.log(response.data);
Another document type we can POST is form data type. For this we need to do a bit more work and use form-data package:
import axios from 'axios';
import FormData from 'form-data';
function makeForm(data){
var bodyFormData = new FormData();
for (let key in data){
bodyFormData.append(key, data[key]);
}
return bodyFormData;
}
const resposne = await axios.post('https://httpbin.org/post', makeForm({'query': 'cats', 'page': 1}));
console.log(response.data);
Axios is smart enough to fill in the required header details (like content-type and content-length) based on the data argument. So, if we're sending an object it'll set Content-Type header to application/json and form data to application/x-www-form-urlencoded - pretty convenient!
As we've covered before our requests must provide some metadata which helps the server to determine what content to return or whether to cooperate with us at all.
Often, this metadata can be used to identify web scrapers and block them, so when scraping we should avoid standing out and mimic a modern web browser.
To start all browsers set User-Agent and Accept headers. To set them in our axios scraper we should create a Client and copy the values from a Chrome web browser:
import axios from 'axios';
const response = await axios.get(
'https://httpbin.org/get',
{headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
}}
);
console.log(response.data);
This will ensure that every request client is making will include these default headers.
For a complete guide on headers in web scraper blocking see our full introduction article.
When scraping we typically want to apply the same configuration to multiple requests like setting those User-Agent headers for every request our scraper is making to avoid being blocked.
Axios comes with a great shortcut that allows to configure default values for all connections:
import axios from 'axios';
const session = axios.create({
headers: {'User-Agent': 'tutorial program'},
timeout: 5000,
proxy: {
host: 'proxy-url',
port: 80,
auth: {username: 'my-user', password: 'my-password'}
}
}
)
const response1 = await session.get('http://httpbin.org/get');
console.log(response1.data);
const response2 = await session.get('http://httpbin.org/get');
console.log(response2.data);
Here we created an instance of axios that will apply custom headers, timeout and proxy settings to every request!
Sometimes when web-scraping we care about persistent connection state. For websites where we need to login or configure preferences like currency or language - cookies are used to do all of that!
Unfortunately, by default axios doesn't support cookie tracking, however it can be enabled via axios-cookiejar-support extension package:
import axios from 'axios';
import { CookieJar } from 'tough-cookie';
import { wrapper } from 'axios-cookiejar-support';
const jar = new CookieJar();
const session = wrapper(axios.create({ jar }));
async function setLocale(){
// set cookies:
let respSetCookies = await session.get('http://httpbin.org/cookies/set/locale/usa');
// retrieve existing cookies:
let respGetCookies = await session.get('http://httpbin.org/cookies');
console.log(respGetCookies.data);
}
setLocale();
In the example above, we're configuring axios instance with a cookie jar object which allows us to have persistent cookies in our web scraping session. If we run this script we should see:
{ cookies: { locale: 'usa' } }
Now that we're familiar HTTP connection and how can we use it in axios HTTP client package let's take a look at the other half of the web scraping process: parsing HTML data!
HTML (HyperText Markup Language) is a text data structure that powers the web. The great thing about it is that it's intended to be machine-readable text content, which is great news for web-scraping as we can easily parse the relevant data with javascript code!
HTML is a tree-type structure that lends easily to parsing. For example, let's take this simple HTML content:
<head>
<title>
</title>
</head>
<body>
<h1>Introduction</h1>
<div>
<p>some description text: </p>
<a class="link" href="http://example.com">example link</a>
</div>
</body>
This is a basic HTML document that a simple website might serve. You can already see the tree-like structure just by indentation of the text, but we can even go further and illustrate it:
This tree structure is brilliant for web scraping as we can easily navigate the whole document and extract specific parts that we want.
For example, to find the title of the website, we can see that it's under <body> and under <h1> element. In other words - if we wanted to extract 1000 titles for 1000 different pages, we would write a rule to find body->h1->text rule, but how do we execute this rule?
When it comes to HTML parsing, there are two standard ways to write these rules: CSS selectors and XPATH selectors. Let's see how can we use them in NodeJS and Cheerio next.
Cheerio is the most popular HTML parsing package in NodeJS which allows us to use CSS selectors to select specific nodes of an HTML tree.
For more on CSS selectors see our complete introduction tutorial which covers basic usage, tips and tricks and common web scraping idioms
To use Cheerio we have to create a tree parser object from an HTML string and then we can use a combination of CSS selectors and element functions to extract specific data:
import cheerio from 'cheerio';
const tree = cheerio.load(`
<head>
<title>My Website</title>
</head>
<body>
<div class="content">
<h1>First blog post</h1>
<p>Just started this blog!</p>
<a href="http://scrapfly.io/blog">Checkout My Blog</a>
</div>
</body>
`);
console.log({
// we can extract text of the node:
title: tree('.content h1').text(),
// or a specific attribute value:
url: tree('.content a').attr('href')
});
In the example above, we're loading Cheerio with our example HTML document and highlighting two ways of selecting relevant data. To select the text of an HTML element we're using text() method and to select a specific attribute we're using the attr() method.
While CSS selectors are short, robust and easy to read sometimes when dealing with complex web pages we might need something more powerful. For that nodeJS also supports XPATH selectors via libraries like xpath and @xmldom/xmldom:
import xpath from 'xpath';
import { DOMParser } from '@xmldom/xmldom'
const tree = new DOMParser().parseFromString(`
<head>
<title>My Website</title>
</head>
<body>
<div class="content">
<h1>First blog post</h1>
<p>Just started this blog!</p>
<a href="http://scrapfly.io/blog">Checkout My Blog</a>
</div>
</body>
`);
console.log({
// we can extract text of the node, which returns `Text` object:
title: xpath.select('//div[@class="content"]/h1/text()', tree)[0].data,
// or a specific attribute value, which return `Attr` object:
url: xpath.select('//div[@class="content"]/a/@href', tree)[0].value,
});
Here, we're replicating our Cheerio example in xmldom + xpath setup selecting title text and the URL's href attribute.
For more on XPATH selectors see our complete introduction tutorial which covers basic usage, tips and tricks and common web scraping idioms
We looked into two methods of parsing HTML content with NodeJS: using CSS selectors with Cheerio and using Xpath selectors with xmldom + xpath. Generally, it's best to stick with Cheerio as it complies with HTML standard better and CSS selectors are easier to work with.
Let's put everything we've learned by exploring an example project next!
We've learned about HTTP connections using axios and HTML parsing using cheerio and now it's time to put everything together and solidify our knowledge.
In this section, we'll write an example scraper for https://www.etsy.com/ which is a user-driven e-commerce website (like Ebay but for crafts). We chose this example to cover two most popular challenges when web scraping with javascript: session cookies and csrf headers.
We'll write a scraper that scrapes the newest products appearing in the vintage product category:
In this example we will be using async await asynchronous programming paradigm. You can read more about it on MDNs official async/await introduction
Let's start off by establishing connection with etsy.com and setting our preferred currency/region:
import cheerio from 'cheerio'
import axios from 'axios';
import { wrapper } from 'axios-cookiejar-support';
import { CookieJar } from 'tough-cookie';
const jar = new CookieJar();
const session = wrapper(
axios.create({
jar: jar,
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
})
);
async function setLocale(currency, region){
let _prewalk = await session.get('https://www.etsy.com/');
let tree = cheerio.load(_prewalk.data);
let csrf = tree('meta[name=csrf_nonce]').attr('content');
try{
let resp = await session.post(
'https://www.etsy.com/api/v3/ajax/member/locale-preferences',
{currency:currency, language:"en-US", region: region},
{headers: {'x-csrf-token': csrf}},
);
}catch (error){
console.log(error);
}
}
await setLocale('USD', 'US');
Here, we are creating an axios instance with cookie tracking support. Then we are connecting to Etsy's homepage and looking for csrf token which allows us to interact with Etsy's backend API. Finally, we're sending preference request to this API which returns some tracking cookies that our cookiejar saves automatically for us.
CSRF token is a special security token used in modern web. It essentially tells the webserver that are continuing our communication and not just randomly popping in somewhere in the middle. In etsy example we started communication by requesting homepage, there we found a token which lets us continue our session. For more on CSRF tokens we recommend this stackoverflow thread
From here, every request we make using our axios instance will include these preference cookies - meaning all of our scrape data will be in USD currency.
With site preferences sorted we can continue with our next step - collect newest product urls in the /vintage/ category:
async function findProducts(category){
let resp = await session.get(
`https://www.etsy.com/c/${category}?explicit=1&category_landing_page=1&order=date_desc`
);
let tree = cheerio.load(resp.data);
return tree('a.listing-link').map(
(i, node) => tree(node).attr('href')
).toArray();
}
console.log(await findProducts('vintage'));
Here, we defined our function which given a category name will return urls from first page. Notice, we've added order=date_desc to sort results in descending order by date to pick up only the latest products.
We're left with implementing product scraping itself:
async function scrapeProduct(url){
let resp = await session.get(url);
let tree = cheerio.load(resp.data);
return {
url: url,
title: tree('h1').text().trim(),
description: tree('p[data-product-details-description-text-content]').text().trim(),
price: tree('div[data-buy-box-region=price] p[class*=title]').text().trim(),
store_url: tree('a[aria-label*="more products from store"]').attr('href').split('?')[0],
images: tree('div[data-component=listing-page-image-carousel] img').map(
(i, node) => tree(node).attr('data-src')
).toArray()
};
}
Similarly to earlier all we're doing in this function is retrieving HTML of the product page and extract product details from the HTML content.
Finally, it's time to put everything together as a runner function:
async function scrapeVintage(){
await setLocale('USD', 'US');
let productUrls = await findProducts('vintage');
return Promise.all(productUrls.map(
(url) => scrapeProduct(url)
))
}
console.log(await scrapeVintage());
Here we're combining all of our defined functions into one scraping task which should produce results like:
[
{
url: 'https://www.etsy.com/listing/688372741/96x125-turkish-oushak-area-rug-vintage?click_key=467d607c570b0d7760a78a00c820a1da4d1e4d0d%3A688372741&click_sum=5f5c2ff9&ga_order=date_desc&ga_search_type=vintage&ga_view_type=gallery&ga_search_query=&ref=sc_gallery-1-1&frs=1&cns=1&sts=1',
title: '9.6x12.5 Turkish Oushak Area Rug, Vintage Wool Rug, Faded Orange Handmade Home Décor, Distressed Blush Beige, Floral Bordered Oriental Rugs',
description: '★ This special rug <...>',
price: '$2,950.00',
store_url: 'https://www.etsy.com/shop/SuffeArt',
images: [
'https://i.etsystatic.com/18572096/r/il/7480a4/3657436348/il_794xN.3657436348_oxay.jpg',
'https://i.etsystatic.com/18572096/r/il/afa2b7/3705052531/il_794xN.3705052531_9xsa.jpg',
'https://i.etsystatic.com/18572096/r/il/dbde4f/3657436290/il_794xN.3657436290_a64r.jpg',
'https://i.etsystatic.com/18572096/r/il/b2002d/3705052595/il_794xN.3705052595_4c7m.jpg',
'https://i.etsystatic.com/18572096/r/il/6ad90d/3705052613/il_794xN.3705052613_kzey.jpg',
'https://i.etsystatic.com/18572096/r/il/ccec83/3705052663/il_794xN.3705052663_1472.jpg',
'https://i.etsystatic.com/18572096/r/il/8be8c9/3657436390/il_794xN.3657436390_5su0.jpg',
'https://i.etsystatic.com/18572096/r/il/c4f65e/3705052709/il_794xN.3705052709_4u9r.jpg',
'https://i.etsystatic.com/18572096/r/il/806141/3705052585/il_794xN.3705052585_fn8p.jpg'
]
},
...
]
Using this example javascript web scraper, we've learned about two important sraping concepts: cookies and headers. We configured currency preferences, learned how to deal with csrf tokens and finally, how to scrape and parse product information!
There are many more challenges in web-scraping, so before we wrap this tutorial up let's take a look at some of them.
Unfortunately identifying nodeJS based web scrapers is really easy which can lead to web scraper blocking.
For more on how scrapers are being identified and blocked see our full introduction article.
To avoid being blocked we can use ScrapFly API which acts as a middleware between client and the server - optimizing all of the requests.
To use ScrapFly all we have to do is to tell axios to send requests through ScrapFly url. We can use this handy wrapper function:
import axios from 'axios';
const ScrapflyOptions = {'key': 'YOUR SCRAPFLY KEY'}
async function getScrapfly(url, options = {}) {
return (await axios.get(
'https://api.scrapfly.io/scrape',
{ params: { ...ScrapflyOptions, ...options, 'url': url } }
)).data;
}
console.log(getScrapfly("http://etsy.com"));
Now, we can we can make requests directly through ScrapFly and take advantage it's many features like javascript rendering which uses a real web browser to fully render a web page:
scrape = await getScrapfly("http://etsy.com/", {render_js: true});
print(scrape.result.content)
Or anti-scraping protection bypass which allows scraping of pages protected by anti-scraping protection services:
scrape = await getScrapfly("http://etsy.com/", {asp: true});
print(scrape.result.content)
We can also configure what proxy type and geographical location ScrapFly requests use:
scrape = await getScrapfly("http://etsy.com/", {country: "US", proxy_pool: "public_residential_pool"});
print(scrape.result.content)
For the full feature set see our full documentation.
To wrap up this tutorial let's take a look at frequently asked questions about web scraping in JS:
Puppeteer is a popular browser automation library for Nodejs. It is frequently used for web scraping. However, we don't always need a web browser to web scrape. In this article, we've learned how can we use Nodejs with a simple HTTP client to scrape web pages. Browsers are very complicated and expensive to run and upkeep so HTTP client web scrapers are much faster and cheaper.
Since NodeJS javascript code is naturally asynchronous we can perform concurrent requests to scrape multiple pages by wrapping a list of scrape promises in Promise.all or Promise.allSettled functions. These async await functions take a list of promise objects and executes them in parallel which can speed up web scraping process hundreds of times:
urls = [...]
async function scrape(url){
...
};
let scrape_promises = urls.map((url) => scrape(url));
await Promise.all(scrape_promises);
When scraping at scale we might need to use proxies to prevent blocking. Most NodeJS http client libraries implement proxy support through simple arguments. For example in axios library we can set proxy using sessions:
const session = axios.create({
proxy: {
host: 'http://111.22.33.44', //proxy ip address with protocol
port: 80, // proxy port
auth: {username: 'proxy-auth-username', password: 'proxy-auth-password'} // proxy auth if needed
}
}
)
Web Scraping with Cheerio and Nodejs is the most popular way to scrape without using browser automation (Puppeteer) and Axios is the most popular way to make HTTP requests. Though less popular alternatives like xmldom shouldn't be overlooked as they can help with scraping more complex web pages.
Since NodeJS engine is not fully browser compliant we cannot automatically click buttons or submit forms. For this something like Puppeteer needs to be used to automate a real web browser. For more see Web Scraping With a Headless Browser: Puppeteer
In this extensive introduction article we've introduced ourselves with NodeJS web scraping ecosystem. We looked into using axios as our HTTP client to collect multiple pages and using cheerio/@xmldom/xmldom to parse information from this data using CSS/XPATH selectors.
Finally, we wrapped everything up with an example nodejs web scraper project which scrapes vintage product information from https://www.etsy.com/ and looked into ScrapFly's middleware solution which takes care of difficult web scraping challenges such as scaling and blocking!
Usually to find scrape targets we look at site search or category pages but there's a better way - sitemaps! In this tutorial, we'll be taking a look at how to find and scrape sitemaps for target locations.
Tutorial on using Node-Unblocker - a nodejs library - to avoid blocking while web scraping and using it to optimize web scraping stacks.
Introduction to using Puppeteer in Nodejs for web scraping dynamic web pages and web apps. Tips and tricks, best practices and example project.
