How to Know What Anti-Bot Service a Website is Using?
In this article we'll take a look at two popular tools: WhatWaf and Wafw00f which can identify what WAF service is used.
"Error 1020: Access Denied" can be seen when web scraping websites powered by Cloudflare WAF. This means Cloudflare has blocked your scraper's IP address.
This error can be caused by a variety of reasons from web scraping too fast to using low-quality proxies. Cloudflare's anti-bot systems are using a variety of technologies to detect web scrapers, like:
So, to avoid this error, the scrapers needs to be fortified against Cloudflare's anti-scraping technologies which can be done by using proxies and better scraping pracitces and libraries.
For more on how Cloudflare is blocking web scrapers see our in-depth explanation blog.
This knowledgebase is provided by Scrapfly — a web scraping API that allows you to scrape any website without getting blocked and implements a dozens of other web scraping conveniences. Check us out 👇