Guide for creating a search engine for any website using web scraping in Python. How to crawl data, index it and display it via js powered GUI.
Web crawling is a form of web scraping with exploration capabilities.
Web scraping tends to focus on specific targets like "a list of product URLs of X e-commerce website" while crawling extends this by adding exploration logic. Crawlers can find all product links in addition to scraping them by exploring the website.
Generally, most web scrapers implement some crawling functionality. For example, if we want to scrape products from an e-commerce website first we'd scrape the page where product URLs are located (like product category) and then scrape each product - this would be a mild form of crawling.
More extreme forms of crawling, like "broad crawling", can crawl and scrape many different website domains and many different web structures. For example, search engines like Google or Bing broad crawl the web and index the contents for search capabilities.