How to Power-Up LLMs with Web Scraping and RAG
In depth look at how to use LLM and web scraping for RAG applications using either LlamaIndex or LangChain.
Modern web scraping often involved a lot of JSON parsing through hidden web data scraping or backend API scraping in particular. There are several ways to parse JSON data in Python.
JMESPath is a popular JSON query language and library available in many languages:
Complete introduction to using JMESPath in Python for JSON parsing and an example web scraping project.
JSONPath is another popular JSON query language and library available in many languages:
Complete introduction to JSONPath and how to use it in Python through an example web scraping project.
Both of these tools are a great way to parse JSON datasets within Python. As for which one is better - generally, JSONPath is more powerful by offering recursive selectors (e.g. $..book will select key book anywhere in the dataset) while Jmespath has a more intuitive syntax and better data reshaping capabilities (e.g. renaming keys and flattening nested data structures).
This knowledgebase is provided by Scrapfly — a web scraping API that allows you to scrape any website without getting blocked and implements a dozens of other web scraping conveniences. Check us out 👇
In depth look at how to use LLM and web scraping for RAG applications using either LlamaIndex or LangChain.
Learn how to scrape forms through a step-by-step guide using HTTP clients and headless browsers.
Learn what minimum advertised price monitoring is and how to apply its concept using Python web scraping.
In this article, we'll explore how to scrape Reddit. We'll extract various social data types from subreddits, posts, and user pages. All of which through plain HTTP requests without headless browser usage.