Web Scraping With Go
Learn web scraping with Golang, from native HTTP requests and HTML parsing to a step-by-step guide to using Colly, the Go web crawling package.
MITM proxy is a proxy server that sits between the client and the server and intercepts or modifies the traffic.
When it comes to web scraping MITM software can be used to inspect web traffic of web browsers and desktop or mobile applications. This information can be used to develop web scrapers that scrape hidden web APIs.
Most commonly MITM software is used in scraping APIs of mobile applications like iOS apps or Android apps. Using MITM public API endpoints can be reverse-engineered and called from web scrapers.
Here are some popular MITM programs used in web scraping:
This knowledgebase is provided by Scrapfly — a web scraping API that allows you to scrape any website without getting blocked and implements a dozens of other web scraping conveniences. Check us out 👇
Learn web scraping with Golang, from native HTTP requests and HTML parsing to a step-by-step guide to using Colly, the Go web crawling package.
In this guide, we'll explore Curlie, a better cURL version. We'll start by defining what Curlie is and how it compares to cURL. We'll also go over a step-by-step guide on using and configuring Curlie to send HTTP requests.
In this article, we'll go over a step-by-step guide on sending and configuring HTTP requests with cURL. We'll also explore advanced usages of cURL for web scraping, such as scraping dynamic pages and avoiding getting blocked.
Learn how to prevent TLS fingerprinting by impersonating normal web browser configurations. We'll start by explaining what the Curl Impersonate is, how it works, how to install and use it. Finally, we'll explore using it with Python to avoid web scraping blocking.