cURL offers an option to create config files for defining command options. These files can be utilized with individual cURL commands to use the configuration defined in them.
Let's use cURL config files. First, create a text file config.txt in the same directory of the cURl terminal and the following data:
# set the HTTP method to POST
-X POST
# set User-Agent
-A "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0"
# add request header
-H "Accept: application/json"
# add cookie values
-b "cookie1=value1; cookie2=value2"
# follow redirects
--location
# add request body
-d "{\"key1\": \"value1\", \"key2\": \"value2\"}" # Enclosed in quotes and escaped internal quotes
Here, we set a few cURL options, such as User-Agent, headers, cookies, HTTP method and request body. To use the cURL config file, we can use the -K cURL config followed by the file name:
curl -K config.txt https://httpbin.dev/anything
The response includes the defined cURL options in the config file:
{
"args": {},
"headers": {
"Accept": [
"application/json"
],
"Accept-Encoding": [
"gzip"
],
"Content-Length": [
"36"
],
"Content-Type": [
"application/x-www-form-urlencoded"
],
"Cookie": [
"cookie1=value1; cookie2=value2"
],
"Host": [
"httpbin.dev"
],
"User-Agent": [
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0"
]
},
"url": "https://httpbin.dev/anything",
"data": "{\"key1\": \"value1\", \"key2\": \"value2\"}",
"files": null,
"form": {
"{\"key1\": \"value1\", \"key2\": \"value2\"}": [
""
]
},
"json": null
}
For more details on cURL, refer to our dedicated guide.
How to Use cURL For Web Scraping
In this article, we'll go over a step-by-step guide on sending and configuring HTTP requests with cURL. We'll also explore advanced usages of cURL for web scraping, such as scraping dynamic pages and avoiding getting blocked.
