Introduction to web scraping with Ruby. How to handle http connections, parse html files for data, best practices, tips and an example project.
Generally, CSS selectors are more brief and popular as it's the same language used to apply styles in web development. However, XPath selectors are more powerful though a bit more complex. In particular XPath offers these 3 powerful features that CSS selectors do not:
When web scraping, it's best to mix both to take advantage of the strengths of these two different tools. For example, let's take a look at this example page and how CSS ad XPath selectors can be used to their strengths:
<div class="product"> <div class="price"> <div data-price="22.84">$22.84</div> </div> <div> <div>Company Name inc.</div> <div> <div>website: <a href="http://example.com">example.com</a></div> </div> </div> </div>
To extract the price we can use a simple CSS selector:
However, CSS selectors cannot find elements by their text value or navigate up the HTML tree. So, to select
Company Name inc. we'd have better luck with XPath:
In the example above we're finding a
div element that contains text
website:, then select its grandparent and the first
div child which is the name of the company!
To summarize XPath vs CSS selectors - both are great tools for parsing HTML though CSS selectors are briefer and easier to use while XPath is more powerful but more verbose and complex. Luckily, most programming languages support both and when web scraping we're best to mix these two technologies!