Web Crawling Knowledgebase

There are 2 ways to determine URL file type: guess by url extension using mimetypes module or do a HTTP HEAD request. Here's how.

When web crawling to avoid non-html pages we can test for page extensions or content types using HEAD requests. Here's how to do it.

Web Scraping and Web Crawling are similar but not quite the same. Crawling is a form of web scraping and here are some major differences.

To find all links in the HTML pages using BeautifulSoup and Python the find_all() method can be used. Here's how to do it.