TooManyRedirects error can be seen when using Python requests module to scrape websites with incorrectly configured redirects:
import requests
requests.get("https://httpbin.dev/redirect/31") # default redirect limit is 30
# will raise:
# TooManyRedirects(requests.exceptions.TooManyRedirects: Exceeded 30 redirects.
# we can set max redirects using requests.Session:
session = requests.Session()
session.max_redirects = 2
session.get("https://httpbin.dev/redirect/3")
When web scraping, this usually means one of 3 things:
- The website is incorrectly configured.
- Our requests are missing important details like headers or cookies.
- The scraper is purposefully redirected in a loop to prevent scraping (i.e. blocking).
To handle ToomanyRedirects exception we should disable automatic redirects and handle them manually:
import requests
session = requests.Session()
response = session.get("https://httpbin.dev/redirect/3", allow_redirects=False)
redirect_url = response.headers['Location']
# now we can manually inspect and fix the redirect url if necessary and then follow it:
response2 = session.get(redirect_url, allow_redirects=False)
