In this article, we’ll take a look at the User-Agent header, what it is and how to use it in web scraping. We'll also generate and rotate user agents to avoid web scraping blocking.
HTTP headers are often presented in varying case though usually in Pacal-Case like
Content-Type. According to HTTP specification header names are case-insensitive, so
content-type is the same as
However, various browsers treat this issue differently. For example, for HTTP1.1 protocol, Chrome and Firefox will show the header name in the same case as it was sent by the server or Pascal-Case. This means when web scraping using HTTP1.1 it's important to replicate the exact case of expected headers to prevent the scraper from being blocked.
For HTTP2+ all headers are required to be lowercase so the scraper should always send headers in lowercase when scraping through HTTP2-capable clients.