How to fix Python requests MissingSchema error?

MissingSchema error can be seen when using Python requests module to scrape invalid URLs without protocol indicator (the http:// bit).

This usually happens when we accidently supply the scraper with relative URLs instead of absolute URLs:

import requests

requests.get("/product/25")  # default redirect limit is 30
# will raise:
# MissingSchema: Invalid URL '/product/10': No scheme supplied. Perhaps you meant http:///product/10?

When web scraping, it's best to always ensure the scraped URLs are absolute using the urljoin() function:

from urllib.parse import urljoin
import requests

response = requests.get("")
urls = [  # lets assume we got this batch of product urls:

for relative_url in urls:
    absolute_url = urljoin(response.url, relative_url)
    # this will result in:
    item_response = requests.get(absolute_url)
Question tagged: requests, Python