Proxy

Proxies are the most crucial part of the modern web scraper. All the traffic will go through this network to give you the ability to change the country and hide your origin to avoid anti-bot solutions.

See in Your Dashboard

Why You Need proxies?

Each proxy is linked to an IP address and each IP address belongs to what we called an Autonomous System Number (ASN). That's how anti bot solutions know the origin of the IP's origin (data center, Internet Service Provider, Business).

Anti-bot solutions will rate-limit the number of requests per IP. This limit is why we need to manage the proxy pool, rotate IPs, and cool them to increase the success rate.

Different kind of proxies exists, by types of IP address issuer:

Datacenter: Cheap but are detected by advanced anti bot solution
Residential: More expensive than datacenter but less detectable than anti bot solution

Our proxy pool is fully managed, cooling, rotation, exclusion based on performance and monitoring. By default, your account comes with two public pools, which are shared with other accounts. For specific client's needs, we can set up a custom proxy pool.

Curl
Pretty HTTP

curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "proxy_pool=public_residential_pool"

"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&proxy_pool=public_residential_pool"

key         = "" 
url         = "https://httpbin.org/anything" 
proxy_pool  = "public_residential_pool"

API Pool Name: public_datacenter_pool

Albania : al
Armenia : am
Argentina : ar
Austria : at
Australia : au
Belgium : be
Bulgaria : bg
Bolivia : bo
Brazil : br
Belarus : by
Canada : ca
Switzerland : ch
Chile : cl
China : cn
Colombia : co
Czechia : cz
Germany : de
Denmark : dk
Ecuador : ec
Estonia : ee
Spain : es
Finland : fi
France : fr
United Kingdom : gb
Georgia : ge
Greece : gr
Croatia : hr
Hungary : hu
Ireland : ie
India : in
Iceland : is
Italy : it
Japan : jp
South Korea : kr
Lithuania : lt
Latvia : lv
Mexico : mx
Netherlands : nl
Norway : no
New Zealand : nz
Peru : pe
Pakistan : pk
Poland : pl
Portugal : pt
Romania : ro
Russia : ru
Saudi Arabia : sa
Sweden : se
Slovakia : sk
Turkey : tr
Ukraine : ua
United States : us

API Pool Name: public_residential_pool

Albania : al
Armenia : am
Argentina : ar
Austria : at
Australia : au
Belgium : be
Bulgaria : bg
Bolivia : bo
Brazil : br
Belarus : by
Canada : ca
Switzerland : ch
Chile : cl
China : cn
Colombia : co
Czechia : cz
Germany : de
Denmark : dk
Ecuador : ec
Estonia : ee
Spain : es
Finland : fi
France : fr
United Kingdom : gb
Georgia : ge
Greece : gr
Croatia : hr
Hungary : hu
Ireland : ie
India : in
Iceland : is
Italy : it
Japan : jp
South Korea : kr
Lithuania : lt
Latvia : lv
Mexico : mx
Netherlands : nl
Norway : no
New Zealand : nz
Peru : pe
Pakistan : pk
Poland : pl
Portugal : pt
Romania : ro
Russia : ru
Saudi Arabia : sa
Sweden : se
Slovakia : sk
Turkey : tr
Ukraine : ua
United States : us

Curl
Pretty HTTP

curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "country=ca"

"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&country=ca"

key      = "" 
url      = "https://httpbin.org/anything" 
country  = "ca"

Proxy Pool

A proxy pool represent a group proxies. By default, proxies are grouped by network type like datacenter, residential, 4g etc. If you request special proxy pool for your need, a proxy pool can represent a set proxies specific to a targeted website or country / region. All available proxy pools are listed in the proxy section on your dashboard. Each proxy pool have it's own cost (mostly due to network type).

By default all accounts have access to these pools:

Public Datacenter Pool: public_datacenter_pool - 1 API calls will be counted
Public Residential Pool: public_residential_pool - 25 API calls will be counted

Related Errors

All related errors are listed below. You can see full description and example of error response on Errors section

ERR::PROXY::NOT_REACHABLE - Proxy was not reachable, it can happened when network issue or proxy itself is in trouble
ERR::PROXY::POOL_NOT_AVAILABLE_FOR_TARGET - The desired proxy pool is not available for the given domain - mostly well known protected domain which require at least residential networks
ERR::PROXY::POOL_NOT_FOUND - Provided Proxy Pool Name do not exists
ERR::PROXY::POOL_UNAVAILABLE_COUNTRY - Country not available for given proxy pool
ERR::PROXY::RESOURCES_SATURATION - Proxy are saturated for the desired country, you can try on other countries. They will come back as soon as possible
ERR::PROXY::TIMEOUT - Proxy do not respond in the given time or was too slow - We have removed it from the pool
ERR::PROXY::UNAVAILABLE - Proxy is unavailable - No proxy available for the target (can be restricted for some website)

Pricing

Each call using the residential proxy pool will count for 25 Scrape API calls.

Curl
Pretty HTTP

curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "country=us"

"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&country=us"

key      = "" 
url      = "https://httpbin.org/anything" 
country  = "us"

API Response contains header X-Scrapfly-Api-Cost indicate you the billed amount