What's the difference between Web Scraping and Crawling?

Web crawling is a form of web scraping with exploration capabilities.
Web scraping tends to focus on specific targets like "a list of product URLs of X e-commerce website" while crawling extends this by adding exploration logic. Crawlers can find all product links in addition to scraping them by exploring the website.

Generally, most web scrapers implement some crawling functionality. For example, if we want to scrape products from an e-commerce website first we'd scrape the page where product URLs are located (like product category) and then scrape each product - this would be a mild form of crawling.

More extreme forms of crawling, like "broad crawling", can crawl and scrape many different website domains and many different web structures. For example, search engines like Google or Bing broad crawl the web and index the contents for search capabilities.

How to Crawl the Web with Python

For more on crawling, see our full introduction tutorial for web crawling with Python

How to Crawl the Web with Python
Question tagged: Web Crawling

Related Posts

How to Scrape Sitemaps to Discover Scraping Targets

Usually to find scrape targets we look at site search or category pages but there's a better way - sitemaps! In this tutorial, we'll be taking a look at how to find and scrape sitemaps for target locations.

Creating Search Engine for any Website using Web Scraping

Guide for creating a search engine for any website using web scraping in Python. How to crawl data, index it and display it via js powered GUI.