Proxy

Introduction

Proxies is the most important part of modern web scraper, all the traffic will go through this network to give you the ability to change the country, hide your origin to avoid anti bot solution.

See in your dashboard

Why you need proxies and how it works ?

Each proxy is linked to an IP address and each IP address belongs to what we called an Autonomous System Number (ASN). That's how anti bot solution know the origin of the IP (datacenter, Internet Service Provider, Business).

While Scraping, systems will rate limit the number of request per IP and that's why we need to manage proxy pool, to rotate IPs and cooling them in order to increase the success rate.

Different kind of proxies exists, by kinds of IP address issuer

• Datacenter: Cheap but are detected by advanced anti bot solution
• Residential: More expensive than datacenter but less detected than anti bot solution
• Mobile (4G): Best proxies for scraping social networks but very expensive

Our proxy pool are fully managed, cooling, rotation, exclusion based on performance and monitoring. By default your account comes with 2 public pools which are shared with other accounts. For specific clients needs we are able to setup a custom proxy pool.

Interactive Example: Residential Proxy Example

Residential Proxy Example Sign in

            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&proxy_pool=public_residential_pool
        

Geo Targeting

• France : fr
• Netherlands : nl
• United Kingdom : gb
• Germany : de
• Italy : it
• Spain : es
• Poland : pl
• United States : us
• Canada : ca
• Brazil : br

Interactive Example: Geo Located Proxy in Canada

Geo Located Proxy in Canada Sign in

            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&country=ca
        

API

By default the API use public_datacenter_pool proxy pool, which add no extra fee. You can specify the desired pool via proxy_pool parameters while calling our API. Example : proxy_pool=public_residential_pool

Available Proxy Pool

• Public Datacenter Pool: public_datacenter_pool
• Public Residential Pool: public_residential_pool

Pricing

Each call using the residential proxy pool will count for 25 Scrape API calls.

Interactive Example: Select United States Location

Select United States Location Sign in

            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&country=us
        

API Response contain header X-Scrapfly-Api-Cost indicate you the billed amount