What are devtools and how they're used in web scraping?

Every modern web browser comes with a special suite of tools for web developers called the Developer Tools (or devtools for short).

This suite contains a lot of powerful tools used in web scraper development that can help to debug and understand how the target websites work.

Devtools can be launched on any website using the F12 key or by right-clicking anywhere on the page and selecting "inspect" option.

devtools inspect right click

To start, the "Elements" tab allows for inspecting the final HTML structure of the page. This can be used to create CSS and XPath selectors for scraping:

devtools elements tab

The "Network" tab (aka the Network Inspector) can be used to inspect the network traffic of the page. This can be used to understand how the website works and discover its backend and hidden APIs:

devtools network inspector tab

One of the most popular features of Network devtools is the ability to export the requests to cURL commands (right click -> copy as curl) that can be converted to scraping code using the cURL to Python tool.

Related Articles

Web Scraping With Ruby

Introduction to web scraping with Ruby. How to handle http connections, parse html files for data, best practices, tips and an example project.

RUBY
HTTP
DATA-PARSING
CSS-SELECTORS
XPATH
INTRO
Web Scraping With Ruby

How to Parse XML

In this article, we'll explain about XML parsing. We'll start by defining XML files, their format and how to navigate them for data extraction.

PYTHON
CSS-SELECTORS
XPATH
DATA-PARSING
How to Parse XML

Web Scraping With NodeJS and Javascript

In this article we'll take a look at scraping using Javascript through NodeJS. We'll cover common web scraping libraries, frequently encountered challenges and wrap everything up by scraping etsy.com

NODEJS
HTTP
DATA-PARSING
CSS-SELECTORS
INTRO
Web Scraping With NodeJS and Javascript

Web Scraping With PHP 101

Introduction to web scraping with PHP. How to handle http connections, parse html files for data, best practices, tips and an example project.

PHP
HTTP
DATA-PARSING
XPATH
INTRO
Web Scraping With PHP 101

Web Scraping with Go

Learn web scraping with Golang, from native HTTP requests and HTML parsing to a step-by-step guide to using Colly, the Go web crawling package.

GOLANG
HTTP
DATA-PARSING
FRAMEWORK
Web Scraping with Go

Ultimate XPath Cheatsheet for HTML Parsing in Web Scraping

Ultimate companion for HTML parsing using XPath selectors. This cheatsheet contains all syntax explanations with interactive examples.

XPATH
DATA-PARSING
Ultimate XPath Cheatsheet for HTML Parsing in Web Scraping