Guide to Axios Headers
Learn about Javascript's Axios headers. How to configure, update, inspect headers in request and responses, how to set defaults and useful tips
Javascript's axios is a popular HTTP client often used when web scraping with nodejs.
To use proxies with axios we can use the proxy
parameter in get
and post
j methods:
const axios = require('axios');
axios.get('https://httpbin.dev/ip', {
proxy: {
host: "160.11.12.13",
port: "8080",
auth: {
username: "username",
password: "password",
}
}
})
.then((response) => {
console.log(response.data);
})
.catch((error) => {
console.error(error);
});
Note that axios
does not support automatic proxy use through terminal environment variables HTTP_PROXY
, HTTPS_PROXY
or ALL_PROXY
.
Axios also does not support SOCKS proxies directly but it can be enabled through socks-proxy-agent library:
const axios = require('axios');
const SocksProxyAgent = require('socks-proxy-agent');
const proxy = 'socks5://localhost:1080';
const httpAgent = new SocksProxyAgent(proxy);
const httpsAgent = new SocksProxyAgent(proxy);
axios.get('http://example.com', {
httpAgent,
httpsAgent,
})
.then((response) => {
console.log(response.data);
})
.catch((error) => {
console.error(error);
});
Finally, when web scraping, it's best to rotate proxies for each request. For that see our article: How to Rotate Proxies in Web Scraping
This knowledgebase is provided by Scrapfly data APIs, check us out! 👇