Python Knowledgebase

What Python libraries support HTTP2?

HTTP2 is still relatively new protocol version that is not yet widely supported. Here are the options for HTTP2 client in Python.
Python httpx vs requests vs aiohttp - key differences

When it comes to these 3 popular http client packages they have different strenghts. Here's how to choose the right fit.
How to use proxies with Python httpx?

To use proxies with Python's httpx library the proxies parameter can be used for http, https and socks5 proxies. Here's how.
How to scrape images from a website?

To scrape all images from a given website python with beautifulsoup and httpx can be used. Here's an example.
How to select dictionary key recursively in Python?

To select dictionary keys recursively in Python the "nested-lookup" package implements the most popular nested key selection algorithms.
What are some ways to parse JSON datasets in Python?

There are several popular options when it comes to JSON dataset parsing in Python. The most popular packages are Jmespath and Jsonpath.
How to use cURL in Python?

cURL through libcurl is a popular library used in HTTP connections and can be used with Python through wrapper libraries like pycurl.
How to open Python http responses in a web browser?

To preview Python http responses we can use temporary files and the built-in webbrowser module. Here's how.

Related

How to scrape HTML table to Excel Spreadsheet (.xlsx)?

To scrape tables to Excel spreadsheet we can use bs4, requets and xlsxwriter packages for Python. Here's how.
How to handle popup dialogs in Playwright?

To handle alert-type pop ups in Playwright the on "dialog" event can be captured and interacted with in both Python and NodeJS playwright clients
How to check if element exists in Playwright?

To check whether an HTML element is present on the page using Playwright the page.locator() method can be used. Here's how.
Selenium: chromedriver executable needs to be in PATH?

selenium error "chromedriver executable needs to be in PATH" means that chrome driver is not installed or reachable - here's how to fix it.
Selenium: geckodriver executable needs to be in PATH?

selenium error "geckodriver executable needs to be in PATH" means that gecko driver is not installed or reachable - here's how to fix it.
How to fix python requests ConnectTimeout error?

Python's ConnectTimeout exception is caused when connection can't be established fast enough. Here's how to fix it.
How to fix Python requests ReadTimeout error?

Python requests.ReadTimeout is caused when resources cannot be read fast enough. Here's how to fix it.
How to fix Python requests MissingSchema error?

Python "requests.MissingSchema" exception is usually caused by a missing protocol part in the URL. Most commonly when relative URL is used.
How to fix Python requests SSLError?

Python's requests.SSLError is caused when encryption certificates mismatch for HTTPS type of URLs. Here's how to fix it.
How to fix Python requests TooManyRedirects error?

Python's requests.TooManyRedirects exception is raised when server continues to redirect >30 times. Here's how to fix it.
How to save and load cookies in Python requests?

To save session between script runs we can save and load requests session cookies to disk. Here's how to do in Python requests.
How to take a screenshot with Playwright?

To take page screenshots in playwright we can use page.screenshot() method. Here's how to select areas and how to screenshot them in playwright.
How to get file type of an URL in Python?

There are 2 ways to determine URL file type: guess by url extension using mimetypes module or do a HTTP HEAD request. Here's how.
How to scroll to an element in Selenium?

In Selenium, the scrollIntoView JavaScript function can be used to scroll to a specific HTML element. Here's how to use it in Selenium.
How to block image loading in Selenium?

To increase Selenium's performance we can block images. To do that with Chrome browser "prefs" launch option can be used. Here's how.

Provided by Scrapfly

This knowledgebase is provided by Scrapfly data APIs, check us out! 👇

Try ScrapFly for FREE!

Related Blog Posts

Everything to Know to Start Web Scraping in Python Today
Everything to Know to Start Web Scraping in Python Today

Complete introduction to web scraping using Python: http, parsing, AI, scaling and deployment.
Guide to Python requests POST method
Guide to Python requests POST method

Discover how to use Python's requests library for POST requests, including JSON, form data, and file uploads, along with response handling tips.
Guide to Python Requests Headers
Guide to Python Requests Headers

Our guide to request headers for Python requests library. How to configure and what do they mean.
What is Parsing? From Raw Data to Insights
What is Parsing? From Raw Data to Insights

Learn about the fundamentals of parsing data, across formats like JSON, XML, HTML, and PDFs. Learn how to use Python parsers and AI models for efficient data extraction.
Concurrency vs Parallelism
Concurrency vs Parallelism

Learn the key differences between Concurrency and Parallelism and how to leverage them in Python and JavaScript to optimize performance in various computational tasks.
How to Track Web Page Changes with Automated Screenshots
How to Track Web Page Changes with Automated Screenshots

In this tutorial we'll take a look at website change tracking using Python, Playwright and Wand. We'll build a tracking tool and schedule it to send us emails on detected changes.
How To Take Screenshots In Python?
How To Take Screenshots In Python?

Learn how to take Python screenshots through Selenium and Playwright, including common browser tips and tricks for customizing web page captures.
How to Power-Up LLMs with Web Scraping and RAG
How to Power-Up LLMs with Web Scraping and RAG

In depth look at how to use LLM and web scraping for RAG applications using either LlamaIndex or LangChain.
How to Scrape Forms
How to Scrape Forms

Learn how to scrape forms through a step-by-step guide using HTTP clients and headless browsers.
How to Build a Minimum Advertised Price (MAP) Monitoring Tool
How to Build a Minimum Advertised Price (MAP) Monitoring Tool

Learn what minimum advertised price monitoring is and how to apply its concept using Python web scraping.
How to Scrape Reddit Posts, Subreddits and Profiles
How to Scrape Reddit Posts, Subreddits and Profiles

In this article, we'll explore how to scrape Reddit. We'll extract various social data types from subreddits, posts, and user pages. All of which through plain HTTP requests without headless browser usage.
How to Scrape With Headless Firefox
How to Scrape With Headless Firefox

Discover how to use headless Firefox with Selenium, Playwright, and Puppeteer for web scraping, including practical examples for each library.
How to Scrape LinkedIn in 2024
How to Scrape LinkedIn in 2024

In this scrape guide we'll be taking a look at one of the most popular web scraping targets - LinkedIn.com. We'll be scraping people profiles, company profiles as well as job listings and search.
Selenium Wire Tutorial: Intercept Background Requests
Selenium Wire Tutorial: Intercept Background Requests

In this guide, we'll explore web scraping with Selenium Wire. We'll define what it is, how to install it, and how to use it to inspect and manipulate background requests.
How to Scrape SimilarWeb Website Traffic Analytics
How to Scrape SimilarWeb Website Traffic Analytics

In this guide, we'll explain how to scrape SimilarWeb through a step-by-step guide. We'll scrape comprehensive website traffic insights, websites comparing data, sitemaps, and trending industry domains.

Company

Tools

Resources

Learn Web Scraping

Usage

© 2024 Scrapfly