Proxy

Introduction
Proxies is the most important part of modern web scraper, all the traffic will go through this network to give you the ability to change the country, hide your origin to avoid anti bot solution.
Why you need proxies and how it works ?
Each proxy is linked to an IP address and each IP address belongs to what we called an Autonomous System Number (ASN). That's how anti bot solution know the origin of the IP (datacenter, Internet Service Provider, Business).
While Scraping, systems will rate limit the number of request per IP and that's why we need to manage proxy pool, to rotate IPs and cooling them in order to increase the success rate.
Different kind of proxies exists, by kinds of IP address issuer
- Datacenter: Cheap but are detected by advanced anti bot solution
- Residential: More expensive than datacenter but less detected than anti bot solution
- Mobile (4G): Best proxies for scraping social networks but very expensive
Our proxy pool are fully managed, cooling, rotation, exclusion based on performance and monitoring. By default your account comes with 2 public pools which are shared with other accounts. For specific clients needs we are able to setup a custom proxy pool.
Interactive Example: Residential Proxy Example
curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&proxy_pool=public_residential_pool
HTTP Call Pretty Print
Geo Targeting
-
France :
fr
-
Netherlands :
nl
-
United Kingdom :
gb
-
Germany :
de
-
Italy :
it
-
Spain :
es
-
Poland :
pl
-
United States :
us
-
Canada :
ca
-
Brazil :
br
Interactive Example: Geo Located Proxy in Canada
curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&country=ca
HTTP Call Pretty Print
API
By default the API use public_datacenter_pool
proxy pool, which add no extra fee. You can specify the desired pool
via proxy_pool
parameters while calling our API. Example : proxy_pool=public_residential_pool
Available Proxy Pool
- Public Datacenter Pool:
public_datacenter_pool
- Public Residential Pool:
public_residential_pool
Pricing
Each call using the residential proxy pool will count for 25 Scrape API calls.
Interactive Example: Select United States Location
curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&country=us
HTTP Call Pretty Print
API Response contain header X-Scrapfly-Api-Cost
indicate you the billed amount