To download a file with cURL, we can use the -O option. Let's attempt to download a PDF file on web-scraping.dev with cURL:
curl -O https://web-scraping.dev/assets/pdf/eula.pdf
The above command will result in the following output and save the file to the same directory of the opened terminal:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10287 100 10287 0 0 10769 0 --:--:-- --:--:-- --:--:-- 10783
Next, let's download a file with cURL to a specific folder location with the --output-dir cURL option:
curl --create-dirs -O --output-dir /eula/pdfs https://web-scraping.dev/assets/pdf/eula.pdf
The above cURL command will download the PDf file and save it to the /eula/pdfs directory. For further details on cURL, refer to our dedicated guide.
Learn how to send HTTP requests with cURL and how to use cURL for web scraping, such as scraping dynamic pages and avoiding getting blocked.
This knowledgebase is provided by Scrapfly — a web scraping API that allows you to scrape any website without getting blocked and implements a dozens of other web scraping conveniences. Check us out 👇
In this guide, we'll explore Curlie, a better cURL version. We'll start by defining what Curlie is and how it compares to cURL. We'll also go over a step-by-step guide on using and configuring Curlie to send HTTP requests.
In this article, we'll go over a step-by-step guide on sending and configuring HTTP requests with cURL. We'll also explore advanced usages of cURL for web scraping, such as scraping dynamic pages and avoiding getting blocked.
Learn how to prevent TLS fingerprinting by impersonating normal web browser configurations. We'll start by explaining what the Curl Impersonate is, how it works, how to install and use it. Finally, we'll explore using it with Python to avoid web scraping blocking.