Proxy
Proxies are the most crucial part of the modern web scraper. All the traffic will go through this network to give you the ability to change the country and hide your origin to avoid anti-bot solutions.
Why You Need proxies?
Each proxy is linked to an IP address and each IP address belongs to what we called an Autonomous System Number (ASN). That's how anti bot solutions know the origin of the IP's origin (data center, Internet Service Provider, Business).
Anti-bot solutions will rate-limit the number of requests per IP. This limit is why we need to manage the proxy pool, rotate IPs, and cool them to increase the success rate.
Different kind of proxies exists, by types of IP address issuer:
- Datacenter: Cheap but are detected by advanced anti bot solution
- Residential: More expensive than datacenter but less detectable than anti bot solution
Our proxy pool is fully managed, cooling, rotation, exclusion based on performance and monitoring. By default, your account comes with two public pools, which are shared with other accounts. For specific client's needs, we can set up a custom proxy pool.
require "uri"
require "net/http"
url = URI("https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fanything&proxy_pool=public_residential_pool")
https = Net::HTTP.new(url.host, url.port);
https.use_ssl = true
request = Net::HTTP::Get.new(url)
response = https.request(request)
puts response.read_body
"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.dev%2Fanything&proxy_pool=public_residential_pool"
"api.scrapfly.io"
"/scrape"
key = ""
url = "https://httpbin.dev/anything"
proxy_pool = "public_residential_pool"
Geo Targeting
public_datacenter_pool
-
Albania :
al -
Armenia :
am -
Argentina :
ar -
Austria :
at -
Australia :
au -
Belgium :
be -
Bulgaria :
bg -
Bolivia :
bo -
Brazil :
br -
Belarus :
by -
Canada :
ca -
Switzerland :
ch -
Chile :
cl -
China :
cn -
Colombia :
co -
Czechia :
cz -
Germany :
de -
Denmark :
dk -
Ecuador :
ec -
Estonia :
ee -
Spain :
es -
Finland :
fi -
France :
fr -
United Kingdom :
gb -
Georgia :
ge -
Greece :
gr -
Croatia :
hr -
Hungary :
hu -
Ireland :
ie -
India :
in -
Iceland :
is -
Italy :
it -
Japan :
jp -
South Korea :
kr -
Lithuania :
lt -
Latvia :
lv -
Mexico :
mx -
Netherlands :
nl -
Norway :
no -
New Zealand :
nz -
Peru :
pe -
Pakistan :
pk -
Poland :
pl -
Portugal :
pt -
Romania :
ro -
Russia :
ru -
Saudi Arabia :
sa -
Sweden :
se -
Slovakia :
sk -
Turkey :
tr -
Ukraine :
ua -
United States :
us
public_residential_pool
-
Albania :
al -
Armenia :
am -
Argentina :
ar -
Austria :
at -
Australia :
au -
Belgium :
be -
Bulgaria :
bg -
Bolivia :
bo -
Brazil :
br -
Belarus :
by -
Canada :
ca -
Switzerland :
ch -
Chile :
cl -
China :
cn -
Colombia :
co -
Czechia :
cz -
Germany :
de -
Denmark :
dk -
Ecuador :
ec -
Estonia :
ee -
Spain :
es -
Finland :
fi -
France :
fr -
United Kingdom :
gb -
Georgia :
ge -
Greece :
gr -
Croatia :
hr -
Hungary :
hu -
Ireland :
ie -
India :
in -
Iceland :
is -
Italy :
it -
Japan :
jp -
South Korea :
kr -
Lithuania :
lt -
Latvia :
lv -
Mexico :
mx -
Netherlands :
nl -
Norway :
no -
New Zealand :
nz -
Peru :
pe -
Pakistan :
pk -
Poland :
pl -
Portugal :
pt -
Romania :
ro -
Russia :
ru -
Saudi Arabia :
sa -
Sweden :
se -
Slovakia :
sk -
Turkey :
tr -
Ukraine :
ua -
United States :
us
require "uri"
require "net/http"
url = URI("https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Ftools.scrapfly.io%2Fapi%2Finfo%2Fip&country=ca")
https = Net::HTTP.new(url.host, url.port);
https.use_ssl = true
request = Net::HTTP::Get.new(url)
response = https.request(request)
puts response.read_body
"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Ftools.scrapfly.io%2Fapi%2Finfo%2Fip&country=ca"
"api.scrapfly.io"
"/scrape"
key = ""
url = "https://tools.scrapfly.io/api/info/ip"
country = "ca"
Proxy Pool
A proxy pool represent a group proxies. By default, proxies are grouped by network type like datacenter, residential, 4g etc. If you request special proxy pool for your need, a proxy pool can represent a set proxies specific to a targeted website or country / region. All available proxy pools are listed in the proxy section on your dashboard. Each proxy pool have it's own cost (mostly due to network type).
By default all accounts have access to these pools:
-
Public Datacenter Pool:
public_datacenter_pool- 1 API Credits will be counted -
Public Residential Pool:
public_residential_pool- 25 API Credits will be counted
Related Errors
All related errors are listed below. You can see full description and example of error response on Errors section
- ERR::PROXY::POOL_NOT_AVAILABLE_FOR_TARGET - The desired proxy pool is not available for the given domain - mostly well known protected domain which require at least residential networks
- ERR::PROXY::POOL_NOT_FOUND - Provided Proxy Pool Name do not exists
- ERR::PROXY::POOL_UNAVAILABLE_COUNTRY - Country not available for given proxy pool
- ERR::PROXY::RESOURCES_SATURATION - Proxy are saturated for the desired country, you can try on other countries. They will come back as soon as possible
- ERR::PROXY::TIMEOUT - Proxy connection was too slow and timeout
- ERR::PROXY::UNAVAILABLE - Proxy is unavailable - The domain (mainly gov website) is restricted, You are using session feature and the proxy is unreachable at the moment
Pricing
Each call using the residential proxy pool will count for 25 Scrape API Credits.
require "uri"
require "net/http"
url = URI("https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Ftools.scrapfly.io%2Fapi%2Finfo%2Fip&proxy_pool=public_residential_pool&country=us")
https = Net::HTTP.new(url.host, url.port);
https.use_ssl = true
request = Net::HTTP::Get.new(url)
response = https.request(request)
puts response.read_body
"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Ftools.scrapfly.io%2Fapi%2Finfo%2Fip&proxy_pool=public_residential_pool&country=us"
"api.scrapfly.io"
"/scrape"
key = ""
url = "https://tools.scrapfly.io/api/info/ip"
proxy_pool = "public_residential_pool"
country = "us"
API Response contains header X-Scrapfly-Api-Cost indicate you the billed amount