Proxy

Proxies are the most crucial part of the modern web scraper. All the traffic will go through this network to give you the ability to change the country and hide your origin to avoid anti-bot solutions.

See in Your Dashboard

Why You Need proxies?

Anti-bot solutions will rate-limit the number of requests per IP. This limit is why we need to manage the proxy pool, rotate IPs, and cool them to increase the success rate.

What are proxies

Each IP address is associated with an Autonomous System Number (ASN) and linked to a proxy. This is how anti-bot solutions track the origin of the IP and limit requests per IP. Our proxy pool rotates IPs, cools them down, and excludes underperforming proxies to increase success rates. We offer two public proxy pools by default, with the option to set up a custom pool. We manage and monitor our proxy pool for optimal performance, with both datacenter and residential proxies available.

Different kind of proxies exists, by types of IP address issuer:

  • Datacenter: Cheap but are detected by advanced anti bot solution
  • Residential: More expensive than datacenter but required for most target protected with anti bot solution

Proxy Pool

A proxy pool represent a group proxies. By default, proxies are grouped by network type like datacenter, residential, 4g etc. One of the main challenges with proxies is the reliability, it's complex to maintain a network of healthy proxies and routing the traffic to them. You don't have to worry about the proxy, simply target the proxy pool and we route the traffic to an healthy proxy meeting your requirements.

All available proxy pools are listed in the proxy section on your dashboard. Each proxy pool have it's own cost (mostly due to network type) and by default these public pools are available:

  • Public Datacenter Pool: public_datacenter_pool - 1 API Credits will be counted
  • Public Residential Pool: public_residential_pool - 25 API Credits will be counted
Sign in 
import requests

url = "https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fanything&proxy_pool=public_residential_pool"

response = requests.request("GET", url)

print(response.text)

# import json
# print(json.loads(response.text)['result']['content'])
# print(json.loads(response.text)['result']['status_code'])

"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.dev%2Fanything&proxy_pool=public_residential_pool"

"api.scrapfly.io"
"/scrape"

key         = "" 
url         = "https://httpbin.dev/anything" 
proxy_pool  = "public_residential_pool"

Geo Targeting

API Pool Name: public_datacenter_pool
  • Albania : al
  • Armenia : am
  • Argentina : ar
  • Austria : at
  • Australia : au
  • Belgium : be
  • Bulgaria : bg
  • Bolivia : bo
  • Brazil : br
  • Belarus : by
  • Canada : ca
  • Switzerland : ch
  • Chile : cl
  • China : cn
  • Colombia : co
  • Czechia : cz
  • Germany : de
  • Denmark : dk
  • Ecuador : ec
  • Estonia : ee
  • Spain : es
  • Finland : fi
  • France : fr
  • United Kingdom : gb
  • Georgia : ge
  • Greece : gr
  • Croatia : hr
  • Hungary : hu
  • Ireland : ie
  • Israel : il
  • India : in
  • Iceland : is
  • Italy : it
  • Japan : jp
  • South Korea : kr
  • Lithuania : lt
  • Latvia : lv
  • Mexico : mx
  • Netherlands : nl
  • Norway : no
  • New Zealand : nz
  • Peru : pe
  • Pakistan : pk
  • Poland : pl
  • Portugal : pt
  • Romania : ro
  • Russia : ru
  • Saudi Arabia : sa
  • Sweden : se
  • Slovakia : sk
  • Türkiye : tr
  • Ukraine : ua
  • United States : us
API Pool Name: public_residential_pool
  • Albania : al
  • Armenia : am
  • Argentina : ar
  • Austria : at
  • Australia : au
  • Belgium : be
  • Bulgaria : bg
  • Bolivia : bo
  • Brazil : br
  • Belarus : by
  • Canada : ca
  • Switzerland : ch
  • Chile : cl
  • China : cn
  • Colombia : co
  • Czechia : cz
  • Germany : de
  • Denmark : dk
  • Ecuador : ec
  • Estonia : ee
  • Spain : es
  • Finland : fi
  • France : fr
  • United Kingdom : gb
  • Georgia : ge
  • Greece : gr
  • Croatia : hr
  • Hungary : hu
  • Ireland : ie
  • Israel : il
  • India : in
  • Iceland : is
  • Italy : it
  • Japan : jp
  • South Korea : kr
  • Lithuania : lt
  • Latvia : lv
  • Mexico : mx
  • Netherlands : nl
  • Norway : no
  • New Zealand : nz
  • Peru : pe
  • Pakistan : pk
  • Poland : pl
  • Portugal : pt
  • Romania : ro
  • Russia : ru
  • Saudi Arabia : sa
  • Sweden : se
  • Slovakia : sk
  • Türkiye : tr
  • Ukraine : ua
  • United States : us
Sign in 
import requests

url = "https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Ftools.scrapfly.io%2Fapi%2Finfo%2Fip&country=ca"

response = requests.request("GET", url)

print(response.text)

# import json
# print(json.loads(response.text)['result']['content'])
# print(json.loads(response.text)['result']['status_code'])

"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Ftools.scrapfly.io%2Fapi%2Finfo%2Fip&country=ca"

"api.scrapfly.io"
"/scrape"

key      = "" 
url      = "https://tools.scrapfly.io/api/info/ip" 
country  = "ca"

All related errors are listed below. You can see full description and example of error response on Errors section

Pricing

Each call using the residential proxy pool will count for 25 Scrape API Credits.

Sign in 
import requests

url = "https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Ftools.scrapfly.io%2Fapi%2Finfo%2Fip&proxy_pool=public_residential_pool&country=us"

response = requests.request("GET", url)

print(response.text)

# import json
# print(json.loads(response.text)['result']['content'])
# print(json.loads(response.text)['result']['status_code'])

"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Ftools.scrapfly.io%2Fapi%2Finfo%2Fip&proxy_pool=public_residential_pool&country=us"

"api.scrapfly.io"
"/scrape"

key         = "" 
url         = "https://tools.scrapfly.io/api/info/ip" 
proxy_pool  = "public_residential_pool" 
country     = "us"
API Response contains header X-Scrapfly-Api-Cost indicate you the billed amount