Knowledge Base
Quick answers to common web scraping questions 161 answers
? Answers
24 answersHow to take screenshots in NodeJS?
Learn how to screenshot in Node.js using Playwright & Puppeteer. Includes installation, concepts, and customization tips.
How to use CSS Selectors in Nim ?
how to parse HTML using CSS selectors in Nim programming language using either CSS3Selectors or nimquery libraries.
What is The cURL (28) Error, Couldn't connect to server?
The cURL (28) indicates a proxy connection error. This error arises when the cURL request can't connect to the proxy server.
How To Use Proxy With cURL?
Proxies are essential to avoid IP address blocking and accessing restricted web pages over a specific location. Learn how to proxies with cURL.
How to Solve the cURL (60) Error When Using Proxy?
The cURL (60) error is a common error encountered when using proxies with cURL. Learn what is the exact cause of this error and how to solve it.
How to Set User Agent With cURL?
The User-Agent header is one of the essential headers which identifies the request sender's device. Learn how to set User-Agent with cURL.
How to Use cURL Config Files?
cURL can be configured using config.txt files which can definite each cURL option. Then, the "-K" option can be used to provide your config.
How to Set cURL Authentication - Full Examples Guide
Learn how to set basic authentication, bearer tokens, and cookie authentication with cURL through a step-by-step guide.
How To Send Multiple cURL Requests in Parallel?
To send request in parallel using cURL command line client the -Z or --parallel option can be used and mixed with other config options.
How to Send a HEAD Request With cURL?
The HEAD HTTP method is used to gather information and metadata about a specific resource. Learn how to send HEAD requests with cURL.
How To Send cURL POST Requests?
POST type requests send data to the web server which is popular http method for web interactions like search. Here's how to POST in cURL.
How to Follow Redirects In cURL?
Redirects are caused by HTTP pages moving to a different location. They can be handled automatically or explicitly - here's how to do it in cURL.
How To Download a File With cURL?
cURL allows for downloading binary files using the cURL -O option here's how to use it effectively and common errors related to file downloads.
How to Copy as cURL With Safari?
Safari allows for capturing HTTP requests on web pages. Learn how to use Safari's developer tools to copy requests as cURL.
How to Copy as cURL With Firefox?
Firefox allows for capturing HTTP requests on web pages. Learn how to use Firefox's developer tools to copy the requests as cURL.
How to Copy as cURL With Edge?
Edge allows for capturing HTTP requests on web pages. Learn how to use Edge's developer tools to copy requests as cURL.
How To Copy as cURL With Google Chrome?
Google Chrome allows for capturing HTTP requests on web pages. Learn how to use Chrome's developer tools to the requests as cURL.
How to Copy as cURL With Brave?
Brave allows for capturing HTTP requests on web pages. Learn how to use brave's developer tools to copy the requests as cURL.
What Python libraries support HTTP2?
HTTP2 is still relatively new protocol version that is not yet widely supported. Here are the options for HTTP2 client in Python.
What case should HTTP headers be in? Lowercase or Pascal-Case?
HTTP header names can be either in lowercase or Pascal-Case and it's important to choose the right case to prevent scraper blocking.
What are some PhantomJS alternatives for automating browsers?
PhantomJS is a popular web browser control and automation tool - here are 3 better modern alternatives.
What are SOCKS5 proxies and how they compare to HTTP proxies?
SOCKS5 is the latest protocol version of SOCKS network routing protocol. Here's how it differs from HTTP.
What are private proxies and how are they used in scraping?
Private proxies mean the proxy is owned by a single user (opposite to shared proxies) which can significantly improve scraping performance.
Mobile vs Residential Proxies - which to choose for scraping?
For web scraping mobile or residential proxies are the best though fill different niches. Here's how to choose.
Ready to scale your web scraping?
Anti-bot bypass, browser rendering, and rotating proxies — all in one API.