How to block resources in Selenium and Python?

To speed up Selenium web scrapers we can block media and other non-essential background requests.

Unfortunately, Selenium by itself doesn't support request interception and blocking so we must use a proxy to handle the blocking for us, then attach this proxy to our Selenium instance.

For example, a popular proxy for such use case is mitproxy. We can easily configure it to block requests by resource type or by resource name.

First install mitmproxy using pip install mitmproxy or package manager available in your operating system. Then, we can create a simple script that will extend mitmproxy with our custom blocking logic:

from mitmproxy import http

# we can block popular 3rd party resources like tracking and advertisements.
  # or something abstract like images
# or block based on resource extension

# this will handle all requests going through proxy:
def request(flow: http.HTTPFlow) -> None:
    url = flow.request.pretty_url
    has_blocked_extension = any(url.endswith(ext) for ext in BLOCK_RESOURCE_EXTENSIONS)
    contains_blocked_key = any(block in url for block in BLOCK_RESOURCE_NAMES)
    if has_blocked_extension or contains_blocked_key:
        print(f"Blocked {url}")
        flow.response = http.Response.make(
            404,  # status code
            b"Blocked",  # content
            {"Content-Type": "text/html"}  # headers

We can run this proxy using mitmproxy -s and it'll start a proxy on localhost:8080 on our machine.

Now, we can attach this proxy to our Selenium instance and it'll block all unwanted requests going through it:

from selenium import webdriver

PROXY = "localhost:8080"  # IP:PORT or HOST:PORT of our mitmproxy

chrome_options = webdriver.ChromeOptions()
# this command enabled proxy for our Selenium browser:
chrome_options.add_argument('--proxy-server=%s' % PROXY)

chrome = webdriver.Chrome(options=chrome_options)
# test it by going to a page with blocked resources:

Using this method to block resources can significantly reduce the bandwidth used by the Selenium scraper - often by 2-10 times! This will also greatly speed up scraping as the browser doesn't need to render unnecessary resources.

🤖 Tip: to use mitmproxy with Selenium and https websites the mitmproxy certificate needs to be installed for that see how to install mitmproxy certificate

Question tagged: Selenium

Related Posts

How to Scrape With Headless Firefox

Discover how to use headless Firefox with Selenium, Playwright, and Puppeteer for web scraping, including practical examples for each library.

Selenium Wire Tutorial: Intercept Background Requests

In this guide, we'll explore web scraping with Selenium Wire. We'll define what it is, how to install it, and how to use it to inspect and manipulate background requests.

Web Scraping Dynamic Web Pages With Scrapy Selenium

Learn how to scrape dynamic web pages with Scrapy Selenium. You will also learn how to use Scrapy Selenium for common scraping use cases, such as waiting for elements, clicking buttons and scrolling.