How To Send Multiple cURL Requests in Parallel?

The cURL -Z or --parallel options are used to pass a list of URLs to request in parallel. For this parallel execution, cURL can handle up to 50 operations at the same time.

cURL also provides two additional parameters for this option:

• --parallel-max
  It specifies the number of connections that cURL can handle simultaneously.
• --parallel-immediate
  If this parameter is used, cURL will prioritize creating new connections instead of waiting for other connections to finish.

Here is how we can send cURL multiple requests with the --parallel option:

curl --parallel --parallel-immediate --parallel-max 5 https://httpbin.dev/headers https://httpbin.dev/headers https://httpbin.dev/headers

The above cURL command will send three requests in parallel to the URL httpbin.dev/headers. It will handle a maximum of 5 simultaneous requests and prioritize creating new connections.

Alternatively, we can specify the URLs to request and pass it to the cURL --config option. First, create a file urls.txt and the URLs:

url = "https://httpbin.dev/headers"
url = "https://httpbin.dev/headers"
url = "https://httpbin.dev/headers"

Next, we'll use the same cURL parallel requests command and pass the config file:

curl --parallel --parallel-immediate --parallel-max 10 --parallel-max 5 --config urls.txt

For more details on cURL, refer to our dedicated guide.

How to Use cURL For Web Scraping

Explore sending and configuring HTTP requests with cURL through a step-by-step guide. You will also explore advanced usages of cURL for web scraping, such as scraping dynamic pages and avoiding getting blocked.