Everything to Know to Start Web Scraping in Python Today
Complete introduction to web scraping using Python: http, parsing, AI, scaling and deployment.
To view Python's HTTP responses in a web browser we can save the contents to a temporary file and open it in the default web browser using Python's webbrowser
module:
import webbrowser
from tempfile import NamedTemporaryFile
# this can work with any response object of any http client like:
import requests
import httpx
def view_in_browser(response):
"""open httpx or requests Response object in default browser"""
# first - save content to a temporary file:
with NamedTemporaryFile("wb", delete=False, suffix=".html") as file:
file.write(response.content)
# open temporary file in a new browser tab as a web page
webbrowser.open_new_tab(f"file://{file.name}")
# - or new window
# webbrowser.open_new(f"file://{file.name}")
# - or current active tab
# webbrowser.open(f"file://{file.name}")
# example use:
response = requests.get("http://scrapfly.io/")
response = httpx.get("http://scrapfly.io/")
view_in_browser(response)
This is a great tool for developing web scrapers as it allows to easily visualize and debug the scraper process as well as to use the browser's developer tools.
This knowledgebase is provided by Scrapfly data APIs, check us out! 👇