How to Set Authentication with cURL - Full Examples Guide

cURl is a viable tool for requesting and transferring data from web pages. However, an obstacle is often encountered: authentication. In this guide, we'll explain how to set cURL authentication for different types:

• Basic auth.
• Bearer tokens.
• Cookies.

Let's get started!

Basic Auth With cURL

The basic auth method is one of the oldest methods for managing authentication. It requires entering basic credentials data, username and password:

Authentication in the above image can be found at httpbin.dev/basic-auth. It requires entering the auth credentials before the request can proceed to the data.

To set basic authentication with cURL, we can use the --user or -u cURL options followed by the --basic option to mark it as basic auth:

curl --user user:passwd --basic https://httpbin.dev/basic-auth/user/passwd

The above cURL command will enter the credentials data (user and passwd) to authorize the request. Here is the response of the above cURL command:

{
  "authorized": true,
  "user": "user"
}

Bearer Tokens With cURL

Bearer tokens are popular authentication methods for modern applications. They are tokens generated after the credentials are entered for the first time and used as a header for the requests to authenticate them.

If we go over the previous authentication endpoint on the browser and refresh the page, we'll find the authentication token used with the request headers:

To set Bearer authentication with cURL, we'll add the above Authorization header to the cURL command using the -H option:

curl -H "Authorization: Basic dXNlcjpwYXNzd2Q=" https://httpbin.dev/basic-auth/user/passwd

From the response, we can see that the cURL request was successfully authenticated:

{
  "authorized": true,
  "user": "user"
}

Authentication Cookies With cURL

Another popular method of managing authentication is using cookies, which function similarly to bearer tokens. After the credentials are validated successfully, a cookie value is created and saved to authorize future requests.

For this example, we'll use the login page on web-scraping.dev:

The above page uses a login form for authentication. If we take a look at the cookies for this domain, we'll find a cookie value for authentication:

To handle cookie authentication with cURL, we'll pass this cookie value to the cURL command in either two ways:

• Adding it as a cookie value using the -b cURL option.
• Adding the cookie value as a header using the -H cURL option.

curl -b "auth=user123-secret-token" https://web-scraping.dev/login
# or
curl -H "cookie: auth=user123-secret-token" https://web-scraping.dev/login

From the response, we can see that the cURL command was authenticated successfully:

  <div class="container text-center">
    <div class="form-text mb-2">Logged in as User123</div>
    <div class="form-text mb-2">The secret message is:<div id="secret-message" class="fw-bold">🤫</div>
    </div>
    <a href="/api/logout"><button type="submit" class="btn btn-primary"> Logout </button> </a>
  </div>

For further details on cURL, including configuring its commands and using it for web scraping, refer to our dedicated guide.

How to Use cURL For Web Scraping

Explore sending and configuring HTTP requests with cURL through a step-by-step guide. You will also explore advanced usages of cURL for web scraping, such as scraping dynamic pages and avoiding getting blocked.