Imperva (aka Incapsula) is a popular WAF service used by many websites like Glassdoor, Udemy and wix.com.
This service is used to block bots, such as web scrapers from accessing the website. So, to scrape public data from these websites, the scrapers need to bypass Imperva Incapsula bot protection.
In this article, we'll explain how to bypass Imperva's anti-scraping protection. We'll start by taking a quick look at what is Imperva, how to identify it and how does it identify web scrapers. Then, we'll take a look at existing techniques and tools for bypassing Imperva bot protection. Let's dive in!
Imperva (previously known as Incapsula) is a WAF service suite that is used to protect websites from unwanted connections. It has legitimate uses in the context of web scraping, which is blocking web scrapers from accessing public data.
Imperva/Incapsula is one of the first WAF services to be used by websites to block web scraping attempts and is generally well understood by the web scraping community. So, let's explain how to identify it and how it's identifying web scrapers.
Most of Imperva bot blocks result in HTTP status codes 400-500, 403 being the most common one. Moreover, blocked pages can appear in status code 200 to confuse web scrapers.
The HTML content often indicates the block is powered by Imperva:
These errors are mostly encountered on the first request to the website. However, it can block the requests at any point during the scrape process as it constantly tracks connections.
Here's the full list of Incapsula block fragments.
Powered By Incapsula text snippet in HTML.Incapsula incident ID keyword in HTML._Incapsula_Resource keyword in HTML.subject=WAF Block Page keyword in HTML.visid_incap value in request headers.X-Iinfo response header.Set-Cookie header has cookie field incap_ses and visid_incap.To detect web scrapers, Imperva is taking advantage of many different analysis and fingerprinting techniques.
Imperva is using combination of the above techniques to establish a unique fingerprint called a trust score for each connecting client.
Based on the final trust score, Imperva decides whether to block the client, let it through or request an additional verification, such as CAPTCHA.
This process seems to be complex and daunting. However, if we study each component individually, we'll find that bypassing Imperva is possible. Let's take a look at each of these components.
TLS (or SSL) fingerprinting is a modern technique of identifying a client based on the way the client and server negotiate an encrypted connection, called as the JA3 fingerprint.
For a secure connection (i.e., https), the encryption method needs to be negotiated between the client and server, as there are many different cipher and encryption options. So, if a connecting client has unusual capabilities, it can be easily identified as a bot.
Libraries used in web scraping can have different encryption capabilities compared to a web browser. So, web scrapers can be easily identified by their TLS fingerprint even before the actual HTTP request is made.
To avoid this, use libraries and tools that are JA3 resistant. To validate this, see ScrapFly's JA3 fingerprint web tool, which shows your connection fingerprint.
For more see our full introduction to TLS fingerprinting which covers TLS fingerprinting in greater detail.
The next step is the IP address analysis. Imperva has access to IP meta information databases that can be used to identify client's intentions and capabilities.
For example, if the IP address belongs to a known proxy or datacenter service, it can be easily identified as a bot and get blocked. If the IP address is from a residential ISP, it is much more likely to be a human. Same for mobile networks.
So, use high-quality residential or mobile proxies to avoid being detected.
For a more in-depth look, see our full introduction to IP blocking and what IP metadata fields are used in bot detection.
With the connection established the next step is HTTP connection analysis.
Most of the natural web runs on HTTP2 and HTTP3 protocols (that's what web browsers prefer). This makes any HTTP1.1 connection suspicious. Moreover, some HTTP libraries still use or default to HTTP1.1, which is a dead giveaway that the request is automated. More modern and feature-rich libraries like Python's httpx or cURL support HTTP2 though it's not enabled by default.
Then, request header values and ordering can be used to identify the client. Web browser headers generation is well understood and reliable, so it's on web scrapers to match it. For example, web browser send headers like User-Agent, Origin and Referer headers and in a specific order to boot.
So, make sure to use HTTP2 and match header values and ordering of a real web browser.
For more see our full introduction to request headers role in blocking
The final step is JavaScript fingerprinting. This is a very powerful technique that can be used to identify a client based on the way it executes JavaScript code.
Since the server is allowed to execute almost any arbitrary javascript code on the client's machine it can extract a lot of information about the client like:
This combination of data can be used by Imperva to identify web scrapers. To get around this, we have two approaches:
The first one is to intercept the JavaScript fingerprinting and feed Imperva with fake data. However, this requires a lot of effort and is not very reliable as any updates to the fingerprinting code will break our logic.
Alternatively, we can use a headless browser to execute the javascript code. This is a much more reliable approach as it's very unlikely that the fingerprinting code will change.
Headless browsers can be controlled by web scraping libraries like Puppeteer, Selenium or Playwright. These tools can be used to control a real web browser to establish a trust-worthy connection with Imperva.
So, using headless browser automation with Selenium, Puppeteer or Playwright is an easy way to handle JavaScript fingerprinting
Many advanced web scraping tools can jump between headless browsers and raw HTTP connections. So, the trust score can be established using a headless browser-based scraping and then switch to fast HTTP requests - this feature is also available in ScrapFly.
Even if we address all of these detection methods, Imperva can still identify scrapers due to continuous behavioral analysis.
As Imperva is tracking all connection details and patterns, it can use this information to adjust the trust score constantly which can lead to blocking or captcha challenges.
So, it's important to distribute the scraping load through multiple agents using proxies and different fingerprint configurations. For example, when scraping using browser automation tools, it's important to use a collection of different profiles like screen size, operating system, rendering capabilities together with IP proxies.
We can see that there's a lot going on when it comes to Imperva's anti-bot technology and since it's using score based approach, we don't necessarily need to bypass all of the detection methods perfectly. To quickly summarize, here's where scrapers can be improved to avoid detection:
Note that as Imperva is developing and improving their methods it's important to stay in touch with web scraping tool and library updates. For example, see Puppeteer stealth plugin for Puppeteer that keeps track of new fingerprinting techniques.
Bypassing Imperva anti-bot while possible is very difficult - let Scrapfly do it for you!
ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale. Each product is equipped with an automatic bypass for any anti-bot system and we achieve this by:
It takes Scrapfly several full-time engineers to maintain this system, so you don't have to!
For example, to scrape pages protected by Imperva using ScrapFly SDK all we need to do is enable the Anti Scraping Protection bypass feature:
from scrapfly import ScrapflyClient, ScrapeConfig
scrapfly = ScrapflyClient(key="YOUR API KEY")
result = scrapfly.scrape(ScrapeConfig(
url="https://www.glassdoor.com/",
asp=True,
# we can also enable headless browsers to render web apps and javascript powered pages
render_js=True,
# and set proxies by country like France
country="FR",
# and proxy type like residential:
proxy_pool=ScrapeConfig.PUBLIC_RESIDENTIAL_POOL,
))
print(result.scrape_result)
To wrap up this article, let's take a look at some frequently asked questions regarding web scraping Imperva protected pages:
Yes. Web scraping publicly available data is perfectly legal around the world as long as the scrapers do not cause damage to the website.
Yes, public page caching services like Google Cache or Archive.org can sometimes be used to bypass Imperva protection as Google and Archive tend to be whitelisted. However, not all pages are cached and the ones that are are often outdated making them unsuitable for web scraping. Cached pages can also be missing parts of content that are loaded dynamically.
Web security is a complex topic so yes but it's not advised as this can be illegal in some countries and in general not sustainable.
There are many other anti-bot WAF services like Cloudflare, Akamai, Datadome, PerimeterX and Kasada, though they function very similarly to Imperva's Incapsula so everything in this tutorial can be applied to them as well.
In this guide, we've taken a look at how to bypass Incapsula (now known as Imperva) when web scraping.
To start, we've taken a look at the detection methods Imperva is using and how can we address each one of them in our scraper code. We saw that using residential proxies and patching common fingerprinting techniques can vastly improve trust scores when it comes to Imperva's bot blocking.
Finally, we've taken a look at some frequently asked questions like alternative bypass methods and the legality of it all.
For an easier way to handle web scraper blocking and power up your web scrapers check out ScrapFly for free!
Discover why you're seeing Cloudflare Error 1015 and learn effective ways to resolve and prevent it.
Understand what causes HTTP 503 errors, when they might indicate blocking, and how to effectively mitigate them.
HTTP 429 is an infamous response code that indicates request throttling or distribution is needed. Let's take a look at how to handle it.