How To Send Multiple cURL Requests in Parallel?

by mostafa Mar 13, 2024

The cURL -Z or --parallel options are used to pass a list of URLs to request in parallel. For this parallel execution, cURL can handle up to 50 operations at the same time.

cURL also provides two additional parameters for this option:

  • --parallel-max
    It specifies the number of connections that cURL can handle simultaneously.
  • --parallel-immediate
    If this parameter is used, cURL will prioritize creating new connections instead of waiting for other connections to finish.

Here is how we can send cURL multiple requests with the --parallel option:

curl --parallel --parallel-immediate --parallel-max 5 https://httpbin.dev/headers https://httpbin.dev/headers https://httpbin.dev/headers

The above cURL command will send three requests in parallel to the URL httpbin.dev/headers. It will handle a maximum of 5 simultaneous requests and prioritize creating new connections.

Alternatively, we can specify the URLs to request and pass it to the cURL --config option. First, create a file urls.txt and the URLs:

url = "https://httpbin.dev/headers"
url = "https://httpbin.dev/headers"
url = "https://httpbin.dev/headers"

Next, we'll use the same cURL parallel requests command and pass the config file:

curl --parallel --parallel-immediate --parallel-max 10 --parallel-max 5 --config urls.txt

For more details on cURL, refer to our dedicated guide.

How to Use cURL For Web Scraping

In this article, we'll go over a step-by-step guide on sending and configuring HTTP requests with cURL. We'll also explore advanced usages of cURL for web scraping, such as scraping dynamic pages and avoiding getting blocked.

How to Use cURL For Web Scraping

Related Articles

Guide to using JSON with cURL

Learn how to send JSON with `cURL` using files, inline data, environment variables, and `jq`. Includes real-world examples for Slack & Google Translate.

CURL
Guide to using JSON with cURL

How to Ignore cURL SSL Errors

Learn to handle SSL errors in cURL, including using self-signed certificates. Explore common issues, safe practices.

CURL
SSL
How to Ignore cURL SSL Errors

How to Use cURL to Download Files

Master file downloads with curl and discover advanced use cases.

CURL
TOOLS
How to Use cURL to Download Files

cURL vs Wget: Key Differences Explained

curl and wget are both popular terminal tools but often used for different tasks - let's take a look at the differences.

CURL
HTTP
TOOLS
cURL vs Wget: Key Differences Explained

How to Use cURL GET Requests

Here's everything you need to know about cURL GET requests and some common pitfalls you should avoid.

CURL
How to Use cURL GET Requests

Sending HTTP Requests With Curlie: A better cURL

In this guide, we'll explore Curlie, a better cURL version. We'll start by defining what Curlie is and how it compares to cURL. We'll also go over a step-by-step guide on using and configuring Curlie to send HTTP requests.

CURL
HTTP
TOOLS
Sending HTTP Requests With Curlie: A better cURL