Congratulations!

You've made it through Scrapfly Academy 1.1 which covers most of educational material Scrapfly produced in the past 2 years. We appreciate all of our readers and before we go, let's take at some key takeaways:

  • Web Scraping can be done almost in any programming language.
  • HTTP clients are used to retrieve web pages.
  • Web Browsers can be used as an HTTP client alternative.
  • HTML parsers are used to parse web pages.
  • Hidden APIs power many web pages which can be scraped.
  • There's hidden data on many pages.
  • JSON is becoming a big part of the web and it can be parsed just like HTML.
  • Scrapers can be blocked because of misconfiguration or detection.

Example Projects

To solidify all of this knowledge, take a look at some of the example projects that you can start reviewing and drawing inspiration from.

In particular, Scrapfly blog is full of hands-on tutorials on how to scrape popular targets like Amazon, Google, Zillow, Instagram etc. This can be found under blog tag.

python icon
github.com/scrapfly-scrapers

All of Scrapfly scrape guide references are maintained and constantly updated on Github.com

As for more novel web scraping examples we've covered various full web scraping projects which can also be found under for specific tasks and niches. Here are some of our favorites:

algolia article icon
Scraping Algolia Search

This project covers how to scrape Algolia search as a service API that powers the search of thousands of websites.

data apis article icon
Real Time Web Scrapers

This project demonstrates how to create real-time web scraping APIs using FastAPI and web scraping.

e-commerce article icon
Observe Market Trends

Introduction to e-commerce scraping - finding products, scheduling and data visualization.

 price tracking article icon
Build a Price Tracker

Hands-on project on how to build a price tracker using Python and web scraping.
< >