Web Scraping With Go
Learn web scraping with Golang, from native HTTP requests and HTML parsing to a step-by-step guide to using Colly, the Go web crawling package.
cURL is a leading HTTP client tool that is used to create HTTP connections. It is powered by a popular C language library libcurl which implements most of the modern HTTP protocol. This includes the newest HTTP features and versions like HTTP3 and IPv6 support and all proxy features.
When it comes to web scraping cURL is the leading library for creating HTTP connections as it supports important features used in web scraping like:
It is used by many web scraping tools and libraries. Many popular HTTP libraries are using libcurl behind the scenes:
However, since cURL is written in C and is incredibly complicated it can be difficult to use in some languages so often loses out to native libraries (like httpx in Python).
This knowledgebase is provided by Scrapfly — a web scraping API that allows you to scrape any website without getting blocked and implements a dozens of other web scraping conveniences. Check us out 👇
Learn web scraping with Golang, from native HTTP requests and HTML parsing to a step-by-step guide to using Colly, the Go web crawling package.
In this guide, we'll explore Curlie, a better cURL version. We'll start by defining what Curlie is and how it compares to cURL. We'll also go over a step-by-step guide on using and configuring Curlie to send HTTP requests.
In this article, we'll go over a step-by-step guide on sending and configuring HTTP requests with cURL. We'll also explore advanced usages of cURL for web scraping, such as scraping dynamic pages and avoiding getting blocked.
Learn how to prevent TLS fingerprinting by impersonating normal web browser configurations. We'll start by explaining what the Curl Impersonate is, how it works, how to install and use it. Finally, we'll explore using it with Python to avoid web scraping blocking.