Proxy

Web Scraping Proxies
Proxies

Proxies are the most crucial part of the modern web scraper. All the traffic will go through this network to give you the ability to change the country and hide your origin to avoid anti-bot solutions.

See in Your Dashboard

Why You Need proxies?

Each proxy is linked to an IP address and each IP address belongs to what we called an Autonomous System Number (ASN). That's how anti bot solutions know the origin of the IP's origin (data center, Internet Service Provider, Business).

Anti-bot solutions will rate-limit the number of requests per IP. This limit is why we need to manage the proxy pool, rotate IPs, and cool them to increase the success rate.

Different kind of proxies exists, by types of IP address issuer:

  • Datacenter: Cheap but are detected by advanced anti bot solution
  • Residential: More expensive than datacenter but less detectable than anti bot solution
  • Mobile (4G): Best proxies for scraping social networks but very expensive

Our proxy pool is fully managed, cooling, rotation, exclusion based on performance and monitoring. By default, your account comes with two public pools, which are shared with other accounts. For specific client's needs, we can set up a custom proxy pool.

Sign in 
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "proxy_pool=public_residential_pool"
"https://api.scrapfly.io/scrape?key=&url=https%253A%252F%252Fhttpbin.org%252Fanything&proxy_pool=public_residential_pool"

key         = "" 
url         = "https://httpbin.org/anything" 
proxy_pool  = "public_residential_pool"

Geo Targeting

  • France : fr
  • Netherlands : nl
  • United Kingdom : gb
  • Germany : de
  • Italy : it
  • Spain : es
  • Poland : pl
  • United States : us
  • Canada : ca
  • Brazil : br
  • Russia : ru
  • Australia : au
  • Mexico : mx
  • Japan : jp
Sign in 
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "country=ca"
"https://api.scrapfly.io/scrape?key=&url=https%253A%252F%252Fhttpbin.org%252Fanything&country=ca"

key      = "" 
url      = "https://httpbin.org/anything" 
country  = "ca"

Proxy Pool

A proxy pool represent a group proxies. By default proxies are grouped by network type like datacenter, residential, 4g etc. If you request special proxy pool for your need, a proxy pool can represent a set proxies specific to a targeted website or country / region. All available proxy pools are listed in the proxy section on your dashboard. Each proxy pool have it's own cost (mostly due to network type).

By default all accounts have access to these pools :

  • Public Datacenter Pool: public_datacenter_pool - No extra cost
  • Public Residential Pool: public_residential_pool - 25 API calls will be counted

All related errors are listed below. You can see full description and example of error response on Errors section

Pricing

Each call using the residential proxy pool will count for 25 Scrape API calls.

Sign in 
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "country=us"
"https://api.scrapfly.io/scrape?key=&url=https%253A%252F%252Fhttpbin.org%252Fanything&country=us"

key      = "" 
url      = "https://httpbin.org/anything" 
country  = "us"
API Response contains header X-Scrapfly-Api-Cost indicate you the billed amount