24.99
""";
Document doc = Jsoup.parse(html, "https://web-scraping.dev/products");
System.out.println(doc.selectFirst("h3 a").text());
System.out.println(doc.selectFirst("h3 a").attr("abs:href"));
System.out.println(doc.selectFirst("div.price").text());
}
}
```
Here is what we got.
```
Box of Chocolate Candy
https://web-scraping.dev/product/1
24.99
```
In the example above we see that jsoup features all parsing features a web scraper might require, like selecting node's text and attribute values!
CSS selectors are by far the most common way to parse HTML documents and they're available in Java through the jsoup package. So, it's the natural next step after fetching the page with OkHttp.
We've covered both CSS and XPATH selectors in great detail previous articles, see our [CSS selector syntax](https://scrapfly.io/blog/posts/parsing-html-with-css) and [XPath selector syntax](https://scrapfly.io/blog/posts/parsing-html-with-xpath) guides.
jsoup is an HTML parsing library so while it's used for web scraping it's not a full web scraping suite.
jsoup HTML parser needs to be paired with HTTP client library like OkHttp (or `Jsoup.connect` for simple scripts) to retrieve HTML pages.
It's likely that these elements are loaded dynamically by javascript. As jsoup doesn't execute javascript it'll not see that content.
When the page needs scripts to run, let's take a look at HtmlUnit next!
[Web Scraping and HTML Parsing with Jsoup and JavaLearn how to harness the power of jsoup, a lightweight and efficient Java library for web scraping and HTML parsing.](https://scrapfly.io/blog/posts/web-scraping-java-jsoup-html-parsing)
## \#3 HtmlUnit
[Headless browsers](https://scrapfly.io/blog/posts/scraping-using-browsers) are becoming very popular in web scraping as a way to deal with dynamic javascript and scraper blocking.
Many websites use complex front-end that generated data on demand either through background requests or javascript functions. To access this data we need a javascript execution environment.
So, to control a headless browser for web scraping in Java we can start with [HtmlUnit](https://www.htmlunit.org/).
[HtmlUnit](https://www.htmlunit.org/) is one of the first major browser automation libraries for Java and it can do almost anything a browser can do.
An `HtmlPage` object is created using `WebClient` with a URL. Then, we click elements, fill in forms and read the page.
The version used here, `5.5.0`, was released in August 2026 under the Apache-2.0 license.
Keep in mind that HtmlUnit runs JavaScript through Rhino, so it doesn't use Chromium.
Here, I pointed HtmlUnit 5.5.0 at web-scraping.dev. Here is what we got.
```
org.htmlunit.ScriptException: missing formal parameter
(https://web-scraping.dev/assets/js/main.js#166)
```
Alternatively, [Playwright](https://playwright.dev/java/) is a modern take on browser automation offering Chromium, Firefox and WebKit, so let's take a look at it next!
## \#4 Playwright for Java
[Playwright](https://playwright.dev/java/) is a cross-platform and cross-language web browser automation toolkit. It's primarily intended to be used as a website test suite but it's perfectly capable of general browser automation and web scraping.
Version `1.63.0` was released in September 2026 under the Apache-2.0 license. Playwright for Java drives Chromium, Firefox and WebKit.
Using Playwright we can automate headless browsers like Firefox or Chrome to navigate the web just like a human being would. We go to URLs, click buttons, write text and execute javascript.
Since Playwright uses a real web browser with javascript environment we can use the browser's HTML parsing capabilities. In Playwright this is implemented through the `locators` feature.
Playwright tracks all of the background requests and responses the browser sends and receives.
In web scraping, we can use this to collect secret data from background responses. For more, see our [hidden APIs](https://scrapfly.io/blog/posts/how-to-scrape-hidden-apis) guide.
Playwright has native support for running multiple browser contexts simultaneously. Each browser context keeps its own cookies and storage.
Playwright is a powerful web scraping tool however it can be difficult to scale up and handle in some web scraping scenarios.
[Selenium](https://www.selenium.dev/documentation/webdriver/) is one of the first major browser automation libraries, so let's take a look at it next!
## \#5 Selenium WebDriver
[Selenium](https://www.selenium.dev/documentation/webdriver/) was initially a tool created to test a website's behavior, but quickly, the demand of web scraping with Selenium has increased.
This tool is quite widespread and is capable of automating different browsers like Chrome, Firefox, Opera and even Internet Explorer through middleware controlled called Selenium WebDriver.
WebDriver is the first browser automation protocol designed by the W3C organization, and it's essentially a middleware protocol service that sits between the client and the browser, translating client commands to web browser actions.
Version `4.50.0` was released on 30 September 2026 under the Apache-2.0 license.
While it's an older protocol, it's still capable and perfectly viable for web scraping.
Scaling up Selenium-powered scrapers can be a difficult challenge. This is where Selenium Grid for web scraping comes in.
Selenium Grid routes sessions to remote browser nodes. It enables running tests in parallel across machines, browser versions and operating systems.
WebDriver BiDi can also stream browser events such as network requests and console messages over a bidirectional connection.
[Playwright vs SeleniumExplore the key differences between Playwright vs Selenium in terms of performance, web scraping, and automation testing for modern web applications.](https://scrapfly.io/blog/posts/playwright-vs-selenium)
For crawling whole websites next, let's take a look at Apache StormCrawler!
Scrapfly
#### Extract structured data automatically?
Scrapfly's Extraction API uses AI to turn any webpage into structured data — no selectors needed.
[Try Free →](https://scrapfly.io/register)## \#6 Apache StormCrawler
Crawling is an extremely useful technique in the data scraping context, as it provides the web data scraper with navigation capabilities.
[Apache StormCrawler](https://github.com/apache/stormcrawler) is a web crawling framework for building continuous crawlers in Java on Apache Storm. Unlike the parsers and browsers above, it can handle a whole stream of URLs instead of a single page.
Its latest version, `3.7.0`, was released on 29 July 2026 under the Apache-2.0 license.
A StormCrawler crawl requires at least one topology. A topology is the core component responsible for managing the entire crawling process, and it doesn't return the pages to us.
Instead, the next essential components are the fetch, parse, status and persistence stages. These are manually wired stages to handle specific events happening during the crawl lifecycle.
So, it's up to us to turn what the stages receive into records, in whatever way suits our pipeline.
Keep in mind that StormCrawler needs a Storm cluster. For a single site scraper a loop and a queue are often enough.
[How to Find All URLs on a DomainLearn how to efficiently find all URLs on a domain using Python and web crawling. Guide on how to crawl entire domain to collect all website data](https://scrapfly.io/blog/posts/how-to-find-all-urls-on-a-domain)
For batch crawls with a crawl database on disk, let's take a look at Apache Nutch next!
## \#7 Apache Nutch
[Apache Nutch](https://nutch.apache.org/) is a popular web crawling framework for large-scale batch data extraction in Java. It earns this name as it's a highly performant, easily accessible and extendible framework.
That being said, we don't have to interact with the underlying Hadoop architecture day to day. From the development perspective, we'll mostly deal with CrawlDb, segments and indexing plugins.
Nutch comes with a crawl store called **CrawlDb**. It holds every known URL and its state.
On the other hand, segments hold fetched and parsed content. Indexing plugins can feed Solr or Elasticsearch, and parsing runs through Apache Tika.
It's essential to visualize this architecture, as it's the core working principle of all Nutch crawls.
Now that we've covered all seven libraries, let's take a look at how to choose between them!
## Choosing Between Java Parsers, Browsers and Crawlers
We've covered seven major Java web scraping libraries - so which one to choose?
Well, it entirely depends on the project you are working on.
I summed up all of the entries here, know what you are dealing with and combine the tools that solves your problems.
One of the most commonly encountered web scraping issues is:
why can't my scraper see the data I see in the web browser?
This is because the content is loaded from a variable using javascript after the page loads. So, the scraper doesn't see the fully rendered page as it's not a web browser capable of loading pages with javascript.
There are a few ways to deal with dynamic javascript-generated content when scraping.
Alternatively, we can **automate a real web browser** to scrape dynamic web pages by integrating it into our web scraper program.
Unfortunately, modern web browsers provide so much information about themselves that they can be easily identified and potentially blocked from accessing a website.
[How to Bypass Anti-Bot Protection When Web ScrapingLearn how anti-bot systems detect scrapers and 5 universal bypass techniques including proxy rotation, fingerprinting, and fortified headless browsers.](https://scrapfly.io/blog/posts/how-to-bypass-anti-bot-protection-when-web-scraping)
## Running These Libraries Against Blocked Pages
Web scraping with OkHttp and jsoup is surprisingly accessible, however once the website blocks the connection or hides the data behind javascript there is not much these two can do. OkHttp stores the block page like any other response. jsoup then parses that, and the selectors come back empty.
ScrapFly is our own product, and this is the part it takes off our plate. ScrapFly is a web scraping API that offers a request middleware service, which picks the proxy, runs the anti-web scraping protection solver and can render the page in a browser. What we get back in Java is the page HTML.
Let's take a look how can we use ScrapFly's API to scrape content with all of these features in Java! There is no ScrapFly package to install. We send a normal request with OkHttp and parse it with jsoup:
java
```java
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
import okhttp3.HttpUrl;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import java.io.IOException;
import java.util.concurrent.TimeUnit;
public class OkHttpExample {
public static void main(String[] args) {
// okhttp gives up after 10s. a render plus bypass often needs longer,
// and the scrapfly docs ask for 155 seconds
OkHttpClient client = new OkHttpClient.Builder()
.readTimeout(155, TimeUnit.SECONDS)
.build();
HttpUrl.Builder urlBuilder = HttpUrl.parse("https://api.scrapfly.io/scrape")
.newBuilder();
// required: the api key, and the page we actually want
urlBuilder.addQueryParameter("key", System.getenv("SCRAPFLY_API_KEY"));
urlBuilder.addQueryParameter("url", "https://web-scraping.dev/products");
// optional:
// anti-bot bypass. older docs call this argument asp
urlBuilder.addQueryParameter("unblocker", "true");
// headless browser, so javascript-rendered markup is in the html
urlBuilder.addQueryParameter("render_js", "true");
// reply with the page itself, not a json wrapper
urlBuilder.addQueryParameter("proxified_response", "true");
// building and send request
String url = urlBuilder.build().toString();
Request request = new Request.Builder()
.url(url)
.build();
try (Response response = client.newCall(request).execute()) {
if (response.isSuccessful()) {
Document doc = Jsoup.parse(
response.body().string(),
"https://web-scraping.dev/products");
System.out.println("Status Code: " + response.code());
System.out.println("cost=" + response.header("X-Scrapfly-Api-Cost"));
System.out.println(doc.selectFirst("div.product h3 a").text());
} else {
System.out.println("Request Failed: " + response.code());
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
```
In the above code, we start by building the API URL with `HttpUrl.Builder`, which handles the URL encoding for us. Then we send the request and pass the body straight to jsoup.
All we have to do is enable the `unblocker` argument (previously `asp`) for the anti-bot bypass, and `render_js` for JavaScript rendering. `proxified_response` matters just as much here. Leave it off and the body is JSON, so the jsoup selector never sees the product node.
For the other parameters, see the [ScrapFly API parameter](https://scrapfly.io/docs/scrape-api/getting-started) list.
For an easier way to handle web scraper blocking, check out [ScrapFly](https://scrapfly.io/register) for free, with 1,000 credits and no card required.
## FAQ
Is Java still good for web scraping in 2026?Yes, Java is usually a strong choice for typed, long-running services and concurrent crawl workers.
Can jsoup execute JavaScript for dynamic pages?No, jsoup is an HTML parsing library so while it's used for web scraping it doesn't execute javascript. For more see our article on [hidden web data](https://scrapfly.io/blog/posts/how-to-scrape-hidden-web-data).
Is Playwright or Selenium better for Java scraping?Well, it entirely depends on the project you are working on. Generally, Selenium has the most educational resources and community support through WebDriver and Grid, but Playwright is quickly gaining popularity in the web scraping community.
How do we call Scrapfly from Java?Just like in the Scrapfly example above - send a request to `api.scrapfly.io/scrape` using OkHttp and set `proxified_response=true` so the response body is HTML for jsoup.
Does HtmlUnit replace a headless browser?No. A headless browser is a browser instance without visible GUI elements. HtmlUnit doesn't use Chromium - see the Rhino error above.
## Summary
In this article, we've covered the 7 best Java libraries for web scraping that cover several steps of the web scraping process, as each of them solves a different problem.
For web scraping connections, we've covered `OkHttp` which is a brilliant, feature-rich HTTP client. On the other hand, there's `Selenium` and `Playwright` which use a whole headless web browser, and `HtmlUnit` which simulates one inside the JVM.
For data parsing, we've taken a look at `jsoup`. For crawling whole websites there's `StormCrawler` for live crawl streams and `Nutch` for big batch crawls.
Finally, for websites that block scrapers or need JavaScript at scale, we've looked at the Scrapfly Web Scraping API.
Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect:
- Do not scrape at rates that could damage the website.
- Do not scrape data that's not available publicly.
- Do not store PII of EU citizens protected by GDPR.
- Do not repurpose *entire* public datasets which can be illegal in some countries.
Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer.
[ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents
Table of Contents
- [Key Takeaways](#key-takeaways)
- [Which Java Web Scraping Library Should We Use?](#which-java-web-scraping-library-should-we-use)
- [\#1 OkHttp](#1-okhttp)
- [\#2 jsoup](#2-jsoup)
- [\#3 HtmlUnit](#3-htmlunit)
- [\#4 Playwright for Java](#4-playwright-for-java)
- [\#5 Selenium WebDriver](#5-selenium-webdriver)
- [\#6 Apache StormCrawler](#6-apache-stormcrawler)
- [\#7 Apache Nutch](#7-apache-nutch)
- [Choosing Between Java Parsers, Browsers and Crawlers](#choosing-between-java-parsers-browsers-and-crawlers)
- [Running These Libraries Against Blocked Pages](#running-these-libraries-against-blocked-pages)
- [FAQ](#faq)
- [Summary](#summary)
Join the Newsletter Get monthly web scraping insights
Scale Your Web Scraping
Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits.
No credit card required 1,000 free API credits Anti-bot bypass included
[Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding)
Not ready? Get our newsletter instead.
## Related Articles
[
python tools
### 7 Top Python Web Scraping Libraries in 2026
Seven Python scraping libraries compared by the parsing, browser, crawling, fetching, JSON-query, and validation jobs th...
](https://scrapfly.io/blog/posts/top-10-web-scraping-libraries-in-python) [
http
### What is HTTP 413 Error? (Payload Too Large)
HTTP status code 413 generally means that POST or PUT data is too large. Let's take a look at how to handle this.
](https://scrapfly.io/blog/posts/http-error-413-payload-too-large) [
http python
### Web Scraping with Python
Introduction tutorial to web scraping with Python. How to collect and parse public data. Challenges, best practices and ...
](https://scrapfly.io/blog/posts/web-scraping-with-python)
Extract structured data with AI, **1,000 free credits** [Start Free](https://scrapfly.io/register)