Imperva (aka Incapsula) is a popular WAF service used by many websites like Glassdoor, Udemy and wix.com.
This service is used to block bots, such as web scrapers from accessing the website. So, to scrape public data from these websites, the scrapers need to bypass Imperva Incapsula bot protection.
In this article, we'll explain how to bypass Imperva's anti-scraping protection. We'll start by taking a quick look at what is Imperva, how to identify it and how does it identify web scrapers. Then, we'll take a look at existing techniques and tools for bypassing Imperva bot protection. Let's dive in!
What is Imperva (aka Incapsula)?
Imperva (previously known as Incapsula) is a WAF service suite that is used to protect websites from unwanted connections. It has legitimate uses in the context of web scraping, which is blocking web scrapers from accessing public data.
Imperva/Incapsula is one of the first WAF services to be used by websites to block web scraping attempts and is generally well understood by the web scraping community. So, let's explain how to identify it and how it's identifying web scrapers.
Imperva Block Page Examples
Most of Imperva bot blocks result in HTTP status codes 400-500, 403 being the most common one. Moreover, blocked pages can appear in status code 200 to confuse web scrapers.
The HTML content often indicates the block is powered by Imperva:
Imperva block page on giffgaff.com website
These errors are mostly encountered on the first request to the website. However, it can block the requests at any point during the scrape process as it constantly tracks connections.
Here's the full list of Incapsula block fragments.
Powered By Incapsula text snippet in HTML.
Incapsula incident ID keyword in HTML.
_Incapsula_Resource keyword in HTML.
subject=WAF Block Page keyword in HTML.
visid_incap value in request headers.
X-Iinfo response header.
Set-Cookie header has cookie field incap_ses and visid_incap.
How does Imperva identify web scrapers?
To detect web scrapers, Imperva is taking advantage of many different analysis and fingerprinting techniques.
Imperva is using combination of the above techniques to establish a unique fingerprint called a trust score for each connecting client.
Based on the final trust score, Imperva decides whether to block the client, let it through or request an additional verification, such as CAPTCHA.
This process seems to be complex and daunting. However, if we study each component individually, we'll find that bypassing Imperva is possible. Let's take a look at each of these components.
TLS Fingerprinting
TLS (or SSL) fingerprinting is a modern technique of identifying a client based on the way the client and server negotiate an encrypted connection, called as the JA3 fingerprint.
For a secure connection (i.e., https), the encryption method needs to be negotiated between the client and server, as there are many different cipher and encryption options. So, if a connecting client has unusual capabilities, it can be easily identified as a bot.
Libraries used in web scraping can have different encryption capabilities compared to a web browser. So, web scrapers can be easily identified by their TLS fingerprint even before the actual HTTP request is made.
To avoid this, use libraries and tools that are JA3 resistant. To validate this, see ScrapFly's JA3 fingerprint web tool, which shows your connection fingerprint.
The next step is the IP address analysis. Imperva has access to IP meta information databases that can be used to identify client's intentions and capabilities.
For example, if the IP address belongs to a known proxy or datacenter service, it can be easily identified as a bot and get blocked. If the IP address is from a residential ISP, it is much more likely to be a human. Same for mobile networks.
So, use high-quality residential or mobile proxies to avoid being detected.
With the connection established the next step is HTTP connection analysis.
Most of the natural web runs on HTTP2 and HTTP3 protocols (that's what web browsers prefer). This makes any HTTP1.1 connection suspicious. Moreover, some HTTP libraries still use or default to HTTP1.1, which is a dead giveaway that the request is automated. More modern and feature-rich libraries like Python's httpx or cURL support HTTP2 though it's not enabled by default.
Then, request header values and ordering can be used to identify the client. Web browser headers generation is well understood and reliable, so it's on web scrapers to match it. For example, web browser send headers like User-Agent, Origin and Referer headers and in a specific order to boot.
So, make sure to use HTTP2 and match header values and ordering of a real web browser.
The final step is JavaScript fingerprinting. This is a very powerful technique that can be used to identify a client based on the way it executes JavaScript code.
Since the server is allowed to execute almost any arbitrary javascript code on the client's machine it can extract a lot of information about the client like:
JavaScript engine details
Hardware and operating system information
Web browser data and rendering capabilities
This combination of data can be used by Imperva to identify web scrapers. To get around this, we have two approaches:
The first one is to intercept the JavaScript fingerprinting and feed Imperva with fake data. However, this requires a lot of effort and is not very reliable as any updates to the fingerprinting code will break our logic.
Alternatively, we can use a headless browser to execute the javascript code. This is a much more reliable approach as it's very unlikely that the fingerprinting code will change.
Headless browsers can be controlled by web scraping libraries like Puppeteer, Selenium or Playwright. These tools can be used to control a real web browser to establish a trust-worthy connection with Imperva.
So, using headless browser automation with Selenium, Puppeteer or Playwright is an easy way to handle JavaScript fingerprinting
Many advanced web scraping tools can jump between headless browsers and raw HTTP connections. So, the trust score can be established using a headless browser-based scraping and then switch to fast HTTP requests - this feature is also available in ScrapFly.
Behavioral Analysis
Even if we address all of these detection methods, Imperva can still identify scrapers due to continuous behavioral analysis.
As Imperva is tracking all connection details and patterns, it can use this information to adjust the trust score constantly which can lead to blocking or captcha challenges.
So, it's important to distribute the scraping load through multiple agents using proxies and different fingerprint configurations. For example, when scraping using browser automation tools, it's important to use a collection of different profiles like screen size, operating system, rendering capabilities together with IP proxies.
How to Bypass Imperva?
We can see that there's a lot going on when it comes to Imperva's anti-bot technology and since it's using score based approach, we don't necessarily need to bypass all of the detection methods perfectly. To quickly summarize, here's where scrapers can be improved to avoid detection:
Use high quality residential or mobile proxies
Use HTTP2 (or later) version for all requests
Match request header values and ordering of a real web browser
Use headless browser automation to generate Javascript fingerprints
Distribute web scraper traffic through multiple agents
Note that as Imperva is developing and improving their methods it's important to stay in touch with web scraping tool and library updates. For example, see Puppeteer stealth plugin for Puppeteer that keeps track of new fingerprinting techniques.
Bypass Imperva with Scrapfly
While bypassing Imperva is possible, maintaining bypass strategies is a lot of work and this is where ScrapFly can help.
Using Scrapfly web scraping API we can defer all of this complexity and bypass logic and focus on web scraping itself!
Scrapfly is not only an Imperva bypasser byt offers many other web scraping quality-of-life features:
For example, to scrape pages protected by Datadome or any other anti-scraping service, when using ScrapFly SDK all we need to do is enable the Anti Scraping Protection bypass feature:
from scrapfly import ScrapflyClient, ScrapeConfig
scrapfly = ScrapflyClient(key="YOUR API KEY")
result = scrapfly.scrape(ScrapeConfig(
url="https://www.glassdoor.com/",
asp=True,
# we can also enable headless browsers to render web apps and javascript powered pages
render_js=True,
# and set proxies by country like France
country="FR",
# and proxy type like residential:
proxy_pool=ScrapeConfig.PUBLIC_RESIDENTIAL_POOL,
))
print(result.scrape_result)
FAQ
To wrap up this article, let's take a look at some frequently asked questions regarding web scraping Imperva protected pages:
Is it legal to scrape Imperva protected pages?
Yes. Web scraping publicly available data is perfectly legal around the world as long as the scrapers do not cause damage to the website.
Is it possible to bypass Imperva using cache services?
Yes, public page caching services like Google Cache or Archive.org can sometimes be used to bypass Imperva protection as Google and Archive tend to be whitelisted. However, not all pages are cached and the ones that are are often outdated making them unsuitable for web scraping. Cached pages can also be missing parts of content that are loaded dynamically.
Is it possible to bypass Imperva entirely and scrape the website directly?
Web security is a complex topic so yes but it's not advised as this can be illegal in some countries and in general not sustainable.
What are some other anti-bot services?
There are many other anti-bot WAF services like Cloudflare, Akamai, Datadome, PerimeterX and Kasada, though they function very similarly to Imperva's Incapsula so everything in this tutorial can be applied to them as well.
Summary
In this guide, we've taken a look at how to bypass Incapsula (now known as Imperva) when web scraping.
To start, we've taken a look at the detection methods Imperva is using and how can we address each one of them in our scraper code. We saw that using residential proxies and patching common fingerprinting techniques can vastly improve trust scores when it comes to Imperva's bot blocking.
Finally, we've taken a look at some frequently asked questions like alternative bypass methods and the legality of it all.
For an easier way to handle web scraper blocking and power up your web scrapers check out ScrapFly for free!
Learn how to prevent TLS fingerprinting by impersonating normal web browser configurations. We'll start by explaining what the Curl Impersonate is, how it works, how to install and use it. Finally, we'll explore using it with Python to avoid web scraping blocking.
In this article, we'll explore the FlareSolverr tool and how to use it to get around Cloudflare while scraping. We'll start by explaining what FlareSolverr is, how it works, how to install and use it. Let's get started!
