How to install mitmproxy certificate on Chrome and Chromium?

mitproxy is a popular man in the middle proxy used in web scraping operations. However, to use it for scraping https websites the custom certificate must be installed.

For Chrome and Chromium browsers follow these steps:

  1. Install mitmproxy using pip install mitmproxy or your systems package manager like:
    • Ubuntu: sudo apt install mitmproxy
    • MacOs: brew install mitmproxy
    • Windows: download binary from mitmproxy.org
  2. Run mitmproxy in the terminal and it'll start a proxy on localhost:8080 on your machine.
  3. Start Chrome instance with mitmproxy proxy attached to it:
    • Linux: google-chrome --proxy-server="localhost:8080"
    • MacOs: open -a "Google Chrome" --args --proxy-server="localhost:8080"
    • Windows: chrome.exe --proxy-server="localhost:8080"
  4. Open http://mitm.it in the browser and download the certificate for your system.
  5. Install the certificate to your Chrome or Chromium browser:
    1. Open chrome://settings/certificates in the browser.
    2. Click on Authorities tab.
    3. Click on Import button and select the certificate from the step 4.
chrome settings to install certificates

Now mitmproxy will be able to intercept and decrypt all https traffic going through it. This enables it being used with headless browser tools like Selenium, Playwright or Puppeteer.

Question tagged: Tools, HTTP

Related Posts

Using API Clients For Web Scraping: Postman

In this article, we'll explore the use of API clients for web scraping. We'll start by explaining how to locate hidden API requests on websites. Then, we'll explore importing, manipulating, and exporting them using Postman to develop efficient API-based web scrapers.

Intro to Parsing HTML and XML with Python and lxml

In this tutorial, we'll take a deep dive into lxml, a powerful Python library that allows for parsing HTML and XML effectively. We'll start by explaining what lxml is, how to install it and using lxml for parsing HTML and XML files. Finally, we'll go over a practical web scraping with lxml.

Use Curl Impersonate to scrape as Chrome or Firefox

Learn how to prevent TLS fingerprinting by impersonating normal web browser configurations. We'll start by explaining what the Curl Impersonate is, how it works, how to install and use it. Finally, we'll explore using it with Python to avoid web scraping blocking.