What are devtools and how they're used in web scraping?

Every modern web browser comes with a special suite of tools for web developers called the Developer Tools (or devtools for short).

This suite contains a lot of powerful tools used in web scraper development that can help to debug and understand how the target websites work.

Devtools can be launched on any website using the F12 key or by right-clicking anywhere on the page and selecting "inspect" option.

devtools inspect right click

To start, the "Elements" tab allows for inspecting the final HTML structure of the page. This can be used to create CSS and XPath selectors for scraping:

devtools elements tab

The "Network" tab (aka the Network Inspector) can be used to inspect the network traffic of the page. This can be used to understand how the website works and discover its backend and hidden APIs:

devtools network inspector tab

One of the most popular features of Network devtools is the ability to export the requests to cURL commands (right click -> copy as curl) that can be converted to scraping code using the cURL to Python tool.

Question tagged: HTTP, Data Parsing, XPath, Css Selectors, Hidden API

Related Posts

Sending HTTP Requests With Curlie: A better cURL

In this guide, we'll explore Curlie, a better cURL version. We'll start by defining what Curlie is and how it compares to cURL. We'll also go over a step-by-step guide on using and configuring Curlie to send HTTP requests.

How to Use cURL For Web Scraping

In this article, we'll go over a step-by-step guide on sending and configuring HTTP requests with cURL. We'll also explore advanced usages of cURL for web scraping, such as scraping dynamic pages and avoiding getting blocked.

Use Curl Impersonate to scrape as Chrome or Firefox

Learn how to prevent TLS fingerprinting by impersonating normal web browser configurations. We'll start by explaining what the Curl Impersonate is, how it works, how to install and use it. Finally, we'll explore using it with Python to avoid web scraping blocking.