Web Scraping With Go
Learn web scraping with Golang, from native HTTP requests and HTML parsing to a step-by-step guide to using Colly, the Go web crawling package.
HTTP headers are often presented in varying case though usually in Pacal-Case like Content-Type. According to HTTP specification header names are case-insensitive, so content-type is the same as Content-Type.
However, various browsers treat this issue differently. For example, for HTTP1.1 protocol, Chrome and Firefox will show the header name in the same case as it was sent by the server or Pascal-Case. This means when web scraping using HTTP1.1 it's important to replicate the exact case of expected headers to prevent the scraper from being blocked.
For HTTP2+ all headers are required to be lowercase so the scraper should always send headers in lowercase when scraping through HTTP2-capable clients.
This knowledgebase is provided by Scrapfly — a web scraping API that allows you to scrape any website without getting blocked and implements a dozens of other web scraping conveniences. Check us out 👇
Learn web scraping with Golang, from native HTTP requests and HTML parsing to a step-by-step guide to using Colly, the Go web crawling package.
In this guide, we'll explore Curlie, a better cURL version. We'll start by defining what Curlie is and how it compares to cURL. We'll also go over a step-by-step guide on using and configuring Curlie to send HTTP requests.
In this article, we'll go over a step-by-step guide on sending and configuring HTTP requests with cURL. We'll also explore advanced usages of cURL for web scraping, such as scraping dynamic pages and avoiding getting blocked.
Learn how to prevent TLS fingerprinting by impersonating normal web browser configurations. We'll start by explaining what the Curl Impersonate is, how it works, how to install and use it. Finally, we'll explore using it with Python to avoid web scraping blocking.