How to Rotate Proxies in Web Scraping
In this article we explore proxy rotation. How does it affect web scraping success and blocking rates and how can we smartly distribute our traffic through a pool of proxies for the best results.
Mobile Proxies is one of the newest type of proxies used in web scraping. These proxies are usually powered either by volunteer mobile phones or special sim card servers. The biggest draw of mobile proxies for web scraping is avoiding of content blocking, throttling or captchas.
As mobile internet infrastructure is more anonymous and mobile user base is very big blocking mobile IPs is much more difficult. That being said, this also means that mobile proxies are much slower, less reliable and generally more expensive than rotating residential or datacenter proxies.
In this article we'll take a look at what we think are the best 5 mobile proxy providers on the market for web scraping, so let's dive in!
We'll be evaluating mobile proxies from the perspective of web scraper so our evaluation will be based on few important details: connection speed and reliability, pricing options and available features.
Before we dive into that though, let's take a quick look at what are mobile proxies, what are their advantages and disadvantages?
Mobile proxies are arguably the best proxies when it comes to web scrapers stealth, however they are also often the worst proxies when it comes to speed and reliability. Meaning supporting bigger scrapers can be quite expensive as bigger proxy pools are needed for more connections/retries.
The most demanded feature of mobile proxies is captcha avoiding. Since mobile connections (3g/4g/5g etc) pool network through cell towers often human users can raise the "bot score" of the entire network thus reducing chances of captcha to the rest of the network.
Main disadvantage here is of course that the host server of the proxy is someones mobile phone or a tablet with an iffy connection to the closest cell tower - meaning the connection is much slower and less reliable. This also means that maintaining a sticky proxy for more than few minutes becomes much more difficult which can make some web scraping very difficult.
To counter this some proxy providers offer mobile proxies through sim card server farms, however this does diminish the stealthiness of the proxies as well as it consolidates a lot of proxy traffic through a single cell tower.
In this article we'll be using this evaluation table to help visualize what to expect from each proxy provider:
Feature | Example Service |
---|---|
Pool Size | How many IPs available in total? |
Geo Targeting | How many countries can IPs be chosen from? |
Anti Bot Bypass | Does this service have anti bot service bypass? |
Javascript Rendering | Does this service provide javascript rendering? |
Log Monitoring | Does this service provide log monitoring dashboard? |
Minimum monthly commitment | What's the minimum monthly cost? |
Price per GB | What's the price per gigabyte of bandwidth used? |
50GB Project Estimate Cost | Estimate of how much an average web scraping project of 50GB bandwidth would cost |
Speed | Response return speed (Above or Below or Close to market average |
Reliability | Rate of connection failures or other errors (Above or Below or Close to market average |
ScrapFly is a web scraping API which can really can be interpreted as a proxy on steroids!
We firmly believe that the future of web-scraping lies beyond simple proxies - a middleware service that does smart proxy selection, solves scraper blocks/captchas and renders javascript is a vital abstraction layer for modern web scrapers.
ScrapFly feels like a proxy but does much more!
Let's take a quick overview of ScrapFly's features and why it makes such a good companion for various size of web scraping projects compared to traditional proxy services that we'll be covering in this article:
To explore these and other offered features see our full documentation!
Let's start by taking a look how ScrapFly itself looks on our evaluation table:
Feature | ScrapFly |
---|---|
Pool Size | 7M IPs |
Geo Targeting | 49+ countries |
Anti Bot Bypass | ✅ |
Javascript Rendering | ✅ |
Log Monitoring | ✅ |
Minimum monthly commitment | $15 |
Price per GB | per request |
50GB Project Estimated Cost | $100/Mo |
Speed | Above market avg |
Reliability | Above market avg |
Oxylabs offers many types of proxies: datacenter, ISP, mobile and residential proxies. It's particularly known for owning one of the biggest proxy pools of 20M+ 3G/4G proxies.
Oxylabs' mobile proxies are known for being a faster and more reliable than market average and it offers some niche features like city based geo-location selection, however this comes at a premium price tag and high minimum commitment:
Feature | Oxylabs |
---|---|
Pool Size | 20M IPs |
Geo Targeting | 82 countries |
Anti Bot Bypass | ❌ |
Javascript Rendering | ❌ |
Log Monitoring | ❌ |
Minimum monthly commitment | $500 |
Price per GB | $25 |
50GB Project Estimated Cost | $1100/Mo |
Speed | Above market avg |
Reliability | Above market avg |
Bright Data (formerly Luminati) is another big name in the proxy world. Like Oxylabs, Bright Data offers many different kinds of proxies like datacenter, rotating residential, ISP and mobile proxies meaning it's good for uncertain projects that might need to adjust proxy needs one month to another.
Let's overview
Feature | Bright Data |
---|---|
Pool Size | 7M IPs |
Geo Targeting | 82 countries |
Anti Bot Bypass | ❌ |
Javascript Rendering | ❌ |
Log Monitoring | ❌ |
Minimum monthly commitment | $300* |
Price per GB | $35 |
50GB Project Estimated Cost | $1400/Mo |
Speed | Above market avg |
Reliability | Above market avg |
* Bright Data also offers "Pay as you go" option which lowers minimum commitment to $60GB/Mo but that's $60/GB of data which is double the market average. Unfortunately, this could only be a viable option for rare extremely low bandwidth scrapers.
Soax is another very similar proxy provider offering residential and mobile (3G/4G) proxies priced by bandwidth. The most attractive feature being that minimum commitment is much lower than of other mobile proxy providers meaning it's a better option for small/medium web scraping projects.
Feature | Soax |
---|---|
Pool Size | 3.5M IPs |
Geo Targeting | 100+ countries |
Anti Bot Bypass | ❌ |
Javascript Rendering | ❌ |
Log Monitoring | ❌ |
Minimum monthly commitment | $75 |
Price per GB | $26 |
50GB Project Estimated Cost | $1300/Mo |
Speed | market avg |
Reliability | Above market avg |
One thing to note that soax' mobile proxies are a bit slower compared to market average but not by much relative to general slow speed of mobile proxies.
Feature | ScrapFly | Oxylabs | Bright Data | Soax |
---|---|---|---|---|
Pool Size (IPs) | 7M | 20M | 7M | 3.5M |
Geo Targeting | 50+ | 82 | 82 | 100+ |
Anti Bot Bypass | ✅ | ❌ | ❌ | ❌ |
Javascript Rendering | ✅ | ❌ | ❌ | ❌ |
Log Monitoring | ✅ | ❌ | ❌ | ❌ |
Minimum monthly commitment | $15 | $500 | $300* | $75 |
Price per GB | per request | $25 | $35 | $26 |
50GB Project Estimate Cost | $100/Mo | $1100/Mo | $1400/Mo | $1300/Mo |
Speed | Above market avg | Above market avg | Above market avg | market avg |
Reliability | Above market avg | Above market avg | Above market avg | Above market avg |
When it comes to modern web scraping a classic proxy service is a tough sell. Even with the recent advances in proxy quality these services still fall short compared to dedicated web scraping APIs which can apply additional, smart connection strategies to prevent captchas, blocking or throttling.
ScrapFly's combination of smart connection strategies and extra UX features like Javascript Rendering and Anti Bot Bypass can make even the hardest targets easily accessible while also simplifying web scraping process!