How to Use cURL GET Requests
Here's everything you need to know about cURL GET requests and some common pitfalls you should avoid.
Proxies are essential to avoid IP address blocking and accessing restricted web pages over a specific location. In a nutshell, proxies are servers that allow for IP address hiding by requesting the web source through another, acting as a middleware.
To add proxy to cURL, we can use the -x
or --proxy
options. Let's explore using these options for proxy types.
The general command for setting HTTP proxy with cURL is the following:
curl -x <protocol>://<proxy_host>:<proxy_port> <url>
The protocol
indicates the HTTP protocol in the case of the HTTP proxies. It can be either htttp
or https
depending on the proxy type, as using the https
protocol with a proxy that doesn't support it will result in errors. The proxy_host
and proxy_port
are configured based on the proxy itself. The proxy_host
can be represented as a numeric IP address or domain name.
Here is an example of setting HTTP proxies with cURL. The actual proxy hostname and port are mocked:
# HTTP proxy
curl --proxy http://123.12.12.12:1234 http://example.com/
curl --proxy http://some_proxy_domain:1234 http://example.com/
# HTTPs proxy
curl --proxy https://123.12.12.12:1234 https://example.com/
curl --proxy https://some_proxy_domain:1234 https://example.com/
To proxy authentication with cURL, we only have to add the username and password as a prefix to the proxy. The same HTTP and HTTPs rules are also applied:
# HTTP proxy with authentication
curl --proxy http://username:password@123.12.12.12:1234 http://example.com/
curl --proxy http://username:password@some_proxy_domain:1234 http://example.com/
# HTTPs proxy with authentication
curl --proxy https://username:password@123.12.12.12:1234 https://example.com/
curl --proxy https://username:password@some_proxy_domain:1234 https://example.com/
The socks proxies operate over the SOCKS4
or the SOCKS5
protocol. Here is how to use socks proxies with cURL:
#SOCKS4
curl --proxy socks4://123.12.12.12:1234 http://example.com
#SOCKS5
curl --proxy socks5://123.12.12.12:1234 http://example.com
This knowledgebase is provided by Scrapfly — a web scraping API that allows you to scrape any website without getting blocked and implements a dozens of other web scraping conveniences. Check us out 👇