How to Power-Up LLMs with Web Scraping and RAG
In depth look at how to use LLM and web scraping for RAG applications using either LlamaIndex or LangChain.
Python is full of great HTTP client libraries but which one is best for web scraping?
By far the most popular choices are httpx, requests and aiohttp - so here are the key differences:
requests - is the oldest and most mature library. It's easy to learn as there are many resources but it doesn't support asyncio or http2aiohttp - is asynchronous take on requests so it fully supports asyncio which can be a major speed boost for web scrapers. Aiohttp also offers a http server making it great for creating web scraping applications that can scrape data and deliver it.httpx - is the new de facto standard when it comes to HTTP clients in Python. It offers vital HTTP2 support and is fully compatible with asyncio making it the best choice for web scraping.For more on how to use HTTPX in web scraping see our hands-on introduction article
This knowledgebase is provided by Scrapfly — a web scraping API that allows you to scrape any website without getting blocked and implements a dozens of other web scraping conveniences. Check us out 👇
In depth look at how to use LLM and web scraping for RAG applications using either LlamaIndex or LangChain.
Learn how to scrape forms through a step-by-step guide using HTTP clients and headless browsers.
Learn what minimum advertised price monitoring is and how to apply its concept using Python web scraping.
In this article, we'll explore how to scrape Reddit. We'll extract various social data types from subreddits, posts, and user pages. All of which through plain HTTP requests without headless browser usage.