Proxies are the most crucial part of the modern web scraper. All the traffic will go through this network to give you the ability to change the country and hide your origin to avoid anti-bot solutions.
Each proxy is linked to an IP address and each IP address belongs to what we called an Autonomous System Number (ASN). That's how anti bot solutions know the origin of the IP's origin (data center, Internet Service Provider, Business).
Anti-bot solutions will rate-limit the number of requests per IP. This limit is why we need to manage the proxy pool, rotate IPs, and cool them to increase the success rate.
Different kind of proxies exists, by types of IP address issuer:
Our proxy pool is fully managed, cooling, rotation, exclusion based on performance and monitoring. By default, your account comes with two public pools, which are shared with other accounts. For specific client's needs, we can set up a custom proxy pool.
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "proxy_pool=public_residential_pool"
"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&proxy_pool=public_residential_pool"
"api.scrapfly.io"
"/scrape"
key = ""
url = "https://httpbin.org/anything"
proxy_pool = "public_residential_pool"
public_datacenter_pool
al
am
ar
at
au
be
bg
bo
br
by
ca
ch
cl
cn
co
cz
de
dk
ec
ee
es
fi
fr
gb
ge
gr
hr
hu
ie
in
is
it
jp
kr
lt
lv
mx
nl
no
nz
pe
pk
pl
pt
ro
ru
sa
se
sk
tr
ua
us
public_residential_pool
al
am
ar
at
au
be
bg
bo
br
by
ca
ch
cl
cn
co
cz
de
dk
ec
ee
es
fi
fr
gb
ge
gr
hr
hu
ie
in
is
it
jp
kr
lt
lv
mx
nl
no
nz
pe
pk
pl
pt
ro
ru
sa
se
sk
tr
ua
us
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "country=ca"
"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&country=ca"
"api.scrapfly.io"
"/scrape"
key = ""
url = "https://httpbin.org/anything"
country = "ca"
A proxy pool represent a group proxies. By default, proxies are grouped by network type like datacenter, residential, 4g etc. If you request special proxy pool for your need, a proxy pool can represent a set proxies specific to a targeted website or country / region. All available proxy pools are listed in the proxy section on your dashboard. Each proxy pool have it's own cost (mostly due to network type).
By default all accounts have access to these pools:
public_datacenter_pool
-
1 API calls will be counted
public_residential_pool
-
25 API calls will be counted
All related errors are listed below. You can see full description and example of error response on Errors section
Each call using the residential proxy pool will count for 25 Scrape API calls.
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "country=us"
"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&country=us"
"api.scrapfly.io"
"/scrape"
key = ""
url = "https://httpbin.org/anything"
country = "us"
API Response contains header X-Scrapfly-Api-Cost
indicate you the billed amount