Learn about Javascript's Axios headers. How to configure, update, inspect headers in request and responses, how to set defaults and useful tips
HTTP Knowledgebase
What case should HTTP headers be in? Lowercase or Pascal-Case?
HTTP header names can be either in lowercase or Pascal-Case and it's important to choose the right case to prevent scraper blocking.
What is Asynchronous Web Scraping?
Asynchronous programming is an accessible way to scale around IO blocking which is especially powerful in web scraping. Here's why.
What are devtools and how they're used in web scraping?
Developer tools suite is used in web development but can also be used in web scraping to understand how target websites work. Here's how to use it.
What is MITM and how is it used in web scraping?
MITM tools can be used to intercept and modify http traffic of various applications like web browser or phone apps in web scraper development.
What is cURL and how is it used in web scraping?
cURL is the most popular HTTP client and library (libcurl) that implements most of HTTP features meaning it's a powerful web scraping tool too.
HTTP vs HTTPS in web scraping ?
HTTPS is a secure version of the HTTP protocol which can complicate the web scraping process in many different ways. Here's what it means.
What is HTTP cookies role in web scraping?
HTTP cookies play a big role in web scraping. They can be used to configure website preferences and play an important role in scraper detection.
Related
Related Blog Posts
Guide to Axios Headers
What is HTTP 401 Error and How to Fix it
Discover the HTTP 401 error meaning, its causes, and solutions in this comprehensive guide. Learn how 401 unauthorized errors occur.
Comprehensive Guide to OkHttp for Java and Kotlin
Learn how to simplify network communication in Java and Android applications using OkHttp.
What is HTTP 407 Status Code and How to Fix it
Learn everything about the HTTP 407 Proxy Authentication Required error. Understand its causes, including misconfigured proxies
Guide to Cloudflare's Error Code 520 and How to Fix it
Quick look at error code 520, what does it mean, its common causes, and how it can be prevented.
What is HTTP 499 Status Code and How to Fix it?
The 499 status code, specific to Nginx, indicates client-canceled requests and can be addressed with retries and optimized timeouts.
Guide to SSL Errors: What do they mean and how to fix them
Overview of SSL errors - what are they, what are common issues and how to resolve them.
What is Error 1015 (Cloudflare) and How to Fix it?
Discover why you're seeing Cloudflare Error 1015 and learn effective ways to resolve and prevent it.
What HTTP Error 412 Precondition Failed and How to Fix it?
Quick look at HTTP status code 412 - what does it mean, its common causes, and how it can be prevented.
What is HTTP Error 503 Service Unavailable and How to Fix it?
Understand what causes HTTP 503 errors, when they might indicate blocking, and how to effectively mitigate them.
Guide to Python requests POST method
Discover how to use Python's requests library for POST requests, including JSON, form data, and file uploads, along with response handling tips.
What is HTTP Error 429 Too Many Request and How to Fix it
HTTP 429 is an infamous response code that indicates request throttling or distribution is needed. Let's take a look at how to handle it.
Guide to Python Requests Headers
Our guide to request headers for Python requests library. How to configure and what do they mean.
Axios vs Fetch: Which HTTP Client to Choose in JS?
Explore the differences between Fetch and Axios - two essential HTTP clients in JavaScript - and discover which is best suited for your project.
What is Status Code 403 Forbidden and How to Fix it
403 Forbidden HTTP status code mean the client is not allowed to view the resources, but why? Let's take a look at reasons and how to bypass it.