[Blog](https://scrapfly.io/blog) / [data-parsing](https://scrapfly.io/blog/tag/data-parsing) / [7 Best C# Web Scraping Libraries in 2026](https://scrapfly.io/blog/posts/best-csharp-web-scraping-libraries) # 7 Best C# Web Scraping Libraries in 2026 by [Ziad Shamndy](https://scrapfly.io/blog/author/ziad) Sep 16, 2026 20 min read [\#data-parsing](https://scrapfly.io/blog/tag/data-parsing) [\#frameworks](https://scrapfly.io/blog/tag/frameworks) [\#tools](https://scrapfly.io/blog/tag/tools) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-csharp-web-scraping-libraries "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-csharp-web-scraping-libraries&text=7%20Best%20C%23%20Web%20Scraping%20Libraries%20in%202026 "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-csharp-web-scraping-libraries "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-csharp-web-scraping-libraries) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-csharp-web-scraping-libraries) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-csharp-web-scraping-libraries) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-csharp-web-scraping-libraries) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fbest-csharp-web-scraping-libraries) Most C# scrapers fail at the category level, not the code level. A team points an HTML parser at a page that builds content in the browser, or launches Chromium for a page the server already renders in full. This guide compares seven C# libraries across three layers of a scraping stack. The layers are HTML parsing, browser automation, and crawl orchestration. Each entry covers the job a library owns, the output it returns, and where it stops. ## Key Takeaways A short summary before the seven entries: - Html Agility Pack is the familiar XPath first parser for tolerant static HTML work. - AngleSharp fits standards oriented DOM work and CSS selector queries. - Playwright for .NET, Selenium WebDriver, and PuppeteerSharp all drive real browsers, with different engine coverage and tooling around them. - Abot and DotnetSpider add crawl orchestration on top of single page retrieval. - No parser and no browser library removes proxy health, blocking, retries, and distributed execution from the application. - Pick the lightest layer that can produce the required fields from the target, then add layers only when the target forces the change. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## Which C# Web Scraping Library Is Best in 2026? There is no single best library, because the seven entries below do not compete for the same job. The practical answer is to choose the lightest layer that can produce the required output from the target page. | Library | Best for | Layer | Selectors | JavaScript | Output | |---|---|---|---|---|---| | Html Agility Pack | Tolerant HTML | Parser | XPath | No | DOM nodes | | AngleSharp | Standards DOM | Parser | CSS | No | DOM objects | | Playwright for .NET | Modern web apps | Browser | Locators | Yes | Page state | | Selenium WebDriver | Cross browser flows | Browser | CSS and XPath | Yes | Page state | | PuppeteerSharp | Chromium tasks | Browser | Locators | Yes | Page and media | | Abot | Crawl control | Crawler | Custom | No | Crawl events | | DotnetSpider | Structured crawls | Crawler | CSS and XPath | No | Data pipeline | Parsing, browser automation, and crawling are three separate jobs. A parser turns one HTML string into queryable nodes. A browser library produces that HTML string for pages the server leaves empty. A crawler decides which URLs get fetched at all. The seven entries follow that same order, starting with the parsing layer and ending with crawl orchestration. Each entry names the output the library hands back, because output shape is what determines the next component in the pipeline. Start with a parser when the response already contains the target fields. ## 1. Html Agility Pack: Best for XPath Based Static HTML Parsing [Html Agility Pack](https://html-agility-pack.net/) loads raw HTML into an `HtmlDocument`, repairs the markup into a node tree, and queries that tree with XPath. Html Agility Pack is a practical fit when an existing scraper already uses XPath extraction rules. Html Agility Pack is tolerant by design. Unclosed tags, stray attributes, and mismatched nesting produce a usable tree instead of a parse exception, which matters because production pages ship broken markup constantly. Three jobs suit Html Agility Pack well: - Parsing server rendered listing and detail pages where every field arrives in the initial response. - Maintaining an existing codebase whose extraction rules are already written as XPath expressions. - Recovering data from pages whose markup is too damaged for a stricter XML parser. Readers new to XPath syntax may want the background before the example below. [Parsing HTML with XpathIntroduction to xpath in the context of web-scraping. How to extract data from HTML documents using xpath, best practices and available tools.](https://scrapfly.io/blog/posts/parsing-html-with-xpath) The example fetches a live product listing and pulls the title and price out of every card on the page. shell```shell dotnet add package HtmlAgilityPack ``` csharp```csharp using HtmlAgilityPack; var http = new HttpClient(); var html = await http.GetStringAsync("https://web-scraping.dev/products"); var document = new HtmlDocument(); document.LoadHtml(html); var cards = document.DocumentNode.SelectNodes("//div[contains(concat(' ', normalize-space(@class), ' '), ' product ')]") ?? Enumerable.Empty(); foreach (var card in cards) { var title = card.SelectSingleNode(".//h3/a")?.InnerText.Trim(); var price = card.SelectSingleNode(".//div[@class='price']")?.InnerText.Trim(); Console.WriteLine($"{title}: {price}"); } ``` `SelectNodes` returns `null` rather than an empty collection when nothing matches, so the null coalescing fallback keeps the loop safe. The price query uses `@class='price'` instead of `contains`, because the substring `price` also matches the surrounding `price-wrap` container. Running the snippet prints the five products on page one, starting with Box of Chocolate Candy at 24.99 and followed by four energy potions at 4.99 each. Html Agility Pack does not execute page JavaScript. Html Agility Pack parses whatever HTML string the application already retrieved, and content that a browser builds after page load never reaches the parser. XPath is one of two selector models available in C#. The next entry covers the other one, along with a DOM that behaves the way a browser DOM behaves. ## 2. AngleSharp: Best for Standards Based DOM and CSS Selectors [AngleSharp](https://anglesharp.github.io/) parses HTML with the HTML5 parsing algorithm and exposes a spec driven DOM. Element traversal, attribute access, and `QuerySelectorAll` behave the way the same calls behave inside a browser. The practical effect is that a frontend developer can reuse selectors straight from browser DevTools. AngleSharp also inserts implied elements the way a browser does, which changes the tree shape for a table written without an explicit `tbody`. Three jobs suit AngleSharp well: - Starting a new C# scraper where CSS selectors are the preferred query language. - Working against modern markup where browser equivalent DOM behavior avoids surprises. - Sharing selector definitions between a browser automation step and a plain parsing step. The example targets the same live listing page used above, so the two parsers can be compared on identical input. shell```shell dotnet add package AngleSharp ``` csharp```csharp using AngleSharp.Html.Parser; var http = new HttpClient(); var html = await http.GetStringAsync("https://web-scraping.dev/products"); var document = await new HtmlParser().ParseDocumentAsync(html); foreach (var card in document.QuerySelectorAll("div.product")) { var title = card.QuerySelector("h3 a")?.TextContent.Trim(); var price = card.QuerySelector("div.price")?.TextContent.Trim(); Console.WriteLine($"{title}: {price}"); } ``` `div.product` matches the class token `product` only, so the query skips the `products` and `products-wrap` containers wrapping the cards. CSS class selectors match whole tokens, which removes the substring trap that XPath `contains` allows. Both snippets return the same five titles and prices. The difference sits in the query language and in how each library repairs unusual markup, not in the data itself. [HtmlAgilityPack vs AngleSharp: Which C# HTML Parser Should You Use?A fixture based comparison of Html Agility Pack and AngleSharp for C# HTML parsing, covering XPath versus CSS selectors, malformed markup, DOM behavior, and when each parser fits a project.](https://scrapfly.io/blog/posts/htmlagilitypack-vs-anglesharp) AngleSharp ships an optional `AngleSharp.Js` extension that runs isolated scripts through a JavaScript interpreter. That extension is not a browser engine, so real rendering, layout, and networked resources still need one of the next three libraries. Both parsers stop at the same boundary. When the server response contains an empty container instead of the target data, the pipeline needs a browser. ## 3. Playwright for .NET: Best for Modern JavaScript Applications [Playwright for .NET](https://playwright.dev/dotnet/) controls Chromium, Firefox, and WebKit through one C# API. The same API covers browser contexts, pages, locators, and waiting. Locators are the part that matters most for scraping. A locator describes an element rather than capturing a handle to one, so Playwright resolves the element at action time and retries while the page is still settling. Three jobs suit Playwright for .NET well: - Extracting data from single page applications where the server returns an application shell. - Driving login flows, filters, and pagination controls that only exist as client side interactions. - Capturing the fully rendered DOM and passing that HTML string to a parser for extraction. The product page below renders its review list from client side data, so the server response contains an empty `#reviews` container. shell```shell dotnet add package Microsoft.Playwright dotnet build pwsh bin/Debug/net8.0/playwright.ps1 install chromium ``` csharp```csharp using Microsoft.Playwright; using var playwright = await Playwright.CreateAsync(); await using var browser = await playwright.Chromium.LaunchAsync(); var page = await browser.NewPageAsync(); await page.GotoAsync("https://web-scraping.dev/product/1"); await page.WaitForSelectorAsync("#reviews .review"); var reviews = await page.Locator("#reviews .review").AllTextContentsAsync(); Console.WriteLine(reviews.Count); foreach (var review in reviews) { Console.WriteLine(review.Trim()); } ``` `WaitForSelectorAsync` blocks until the first review element exists in the DOM, which is the step a plain HTTP request cannot replace. `AllTextContentsAsync` then returns every matching element's text in one call, printing the five reviews rendered on page one. The `playwright.ps1` install step is a one time browser download and requires PowerShell. Replace `net8.0` in the path with whatever target framework the project builds against. [Web Scraping With Playwright in 2026: A Python GuideScrape JavaScript-rendered pages with Playwright and Python in 2026: locators, waiting, request interception, resource blocking, and concurrency.](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python) Playwright's locator model re-resolves elements after DOM updates and auto-waits before actions. Teams with an existing WebDriver investment usually reach for a different library first. ## 4. Selenium WebDriver: Best for Cross Browser WebDriver Workflows [Selenium WebDriver](https://www.selenium.dev/documentation/webdriver/) controls browsers through the W3C WebDriver protocol. A driver process sits between the C# code and the browser, which is the model most QA and automation teams already run. Selenium supports both CSS selectors and XPath through the `By` class, so extraction rules written for either parser above port across with little rewriting. Selenium Manager, included since Selenium 4.6, resolves and downloads the matching driver automatically. Three jobs suit Selenium WebDriver well: - Running the same scraping flow across Chrome, Firefox, Edge, and Safari. - Extending an existing Selenium suite instead of introducing a second automation stack. - Reusing WebDriver knowledge that a team already has in production. The snippet below reads the same client rendered review list, using an explicit wait instead of Playwright's built in waiting. shell```shell dotnet add package Selenium.WebDriver dotnet add package Selenium.Support ``` csharp```csharp using OpenQA.Selenium; using OpenQA.Selenium.Chrome; using OpenQA.Selenium.Support.UI; var options = new ChromeOptions(); options.AddArgument("--headless=new"); using var driver = new ChromeDriver(options); driver.Navigate().GoToUrl("https://web-scraping.dev/product/1"); var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10)); wait.Until(d => d.FindElements(By.CssSelector("#reviews .review")).Count > 0); foreach (var review in driver.FindElements(By.CssSelector("#reviews .review"))) { Console.WriteLine(review.Text.Trim()); } ``` `WebDriverWait` polls until at least one review element exists, because `FindElements` returns an empty list rather than throwing when nothing matches yet. Disposing the driver at the end of the `using` scope closes the browser and stops the driver process. Driver and browser lifecycle stay a Selenium concern. Anti-bot behavior stays an application concern, since a WebDriver session carries its own detectable properties regardless of which browser runs underneath. [Playwright vs SeleniumExplore the key differences between Playwright vs Selenium in terms of performance, web scraping, and automation testing for modern web applications.](https://scrapfly.io/blog/posts/playwright-vs-selenium) Selenium supports Chrome, Firefox, Edge, and Safari. The next library moves in the opposite direction and specializes in one. ## 5. PuppeteerSharp: Best for Chromium Focused Automation [PuppeteerSharp](https://www.puppeteersharp.com/) is a .NET port of the Node.js Puppeteer API. PuppeteerSharp speaks the Chrome DevTools Protocol directly, which makes Chromium level control the reason to pick it. The library covers navigation, locators, waiting, JavaScript evaluation, screenshots, and PDF generation. Recent versions also added Firefox support through WebDriver BiDi, though the Chromium path remains the well travelled one. Three jobs suit PuppeteerSharp well: - Producing screenshots or PDFs of rendered pages as a first class output. - Porting an existing Node.js Puppeteer script into a C# service with minimal redesign. - Working close to the DevTools Protocol for request interception and low level page control. The snippet renders the same product page, counts the review elements, and saves both a screenshot and a PDF. shell```shell dotnet add package PuppeteerSharp ``` csharp```csharp using PuppeteerSharp; await new BrowserFetcher().DownloadAsync(); await using var browser = await Puppeteer.LaunchAsync(new LaunchOptions { Headless = true }); await using var page = await browser.NewPageAsync(); await page.GoToAsync("https://web-scraping.dev/product/1"); await page.WaitForSelectorAsync("#reviews .review"); var reviews = await page.QuerySelectorAllAsync("#reviews .review"); Console.WriteLine(reviews.Length); await page.ScreenshotAsync("product-1.png"); await page.PdfAsync("product-1.pdf"); ``` `BrowserFetcher().DownloadAsync()` downloads a matching Chromium build on first run and reuses the cached copy afterwards. `PdfAsync` works only in headless Chromium, which is why the media output belongs in this entry rather than the Selenium one. Choosing between PuppeteerSharp and Playwright for .NET comes down to ecosystem and browser requirements. Pick PuppeteerSharp for Chromium specific work or an existing Puppeteer codebase, and pick Playwright when Firefox and WebKit coverage matter. [Puppeteer Stealth: Complete Guide to Avoiding DetectionComplete guide to puppeteer-extra-plugin-stealth for avoiding bot detection. Learn how detection works, configure stealth evasion modules, implement complementary techniques, and scale with cloud browsers.](https://scrapfly.io/blog/posts/puppeteer-stealth-complete-guide) The first five libraries all operate on one page at a time. The final two decide which pages get requested in the first place. Scrapfly #### Extract structured data automatically? Scrapfly's Extraction API uses AI to turn any webpage into structured data — no selectors needed. [Try Free →](https://scrapfly.io/register)## 6. Abot: Best for Event Driven Site Crawling [Abot](https://github.com/sjdirect/abot) is a C# crawler framework that handles threading, scheduling, link parsing, and politeness rules. The application subscribes to crawl events and processes page data as each page completes. The NuGet package is named `Abot`, and the version 2 series targets .NET Standard 2.0 with namespaces under `Abot2`. The most recent release on NuGet is 2.0.70, published in August 2021. Treat Abot as stable rather than actively evolving, and check the repository before committing a long lived project to it. Three jobs suit Abot well: - Discovering and processing many linked pages under explicit crawl limits. - Enforcing per domain delays and robots rules across a whole crawl. - Plugging custom decision logic into the crawl without rewriting the scheduler. The crawl below stays inside the demo store, respects the site's declared crawl delay, and reads each page with the AngleSharp document Abot attaches to every crawled page. shell```shell dotnet add package Abot dotnet add package AngleSharp ``` csharp```csharp using AngleSharp.Dom; using Abot2.Crawler; using Abot2.Poco; var config = new CrawlConfiguration { MaxPagesToCrawl = 10, MinCrawlDelayPerDomainMilliSeconds = 2000, IsRespectRobotsDotTextEnabled = true, UserAgentString = "csharp-crawler-demo/1.0" }; var crawler = new PoliteWebCrawler(config); var seen = 0; crawler.PageCrawlCompleted += (sender, e) => { var crawled = e.CrawledPage; var status = crawled.HttpResponseMessage?.StatusCode; var title = crawled.AngleSharpHtmlDocument?.QuerySelector("title")?.TextContent.Trim(); seen++; Console.WriteLine($"[{status}] {crawled.Uri.AbsoluteUri} {title}"); }; await crawler.CrawlAsync(new Uri("https://web-scraping.dev/products")); Console.WriteLine($"Pages crawled: {seen}"); ``` `PageCrawlCompleted` fires once per finished page, and `AngleSharpHtmlDocument` gives the handler a parsed DOM without a second parse step. Setting `MinCrawlDelayPerDomainMilliSeconds` to 2000 matches the crawl delay the target's robots.txt declares. Abot is not a browser. Abot issues HTTP requests and parses the responses, so a page that renders its content client side arrives at the handler with that content missing. [How to Find All URLs on a DomainLearn how to efficiently find all URLs on a domain using Python and web crawling. Guide on how to crawl entire domain to collect all website data](https://scrapfly.io/blog/posts/how-to-find-all-urls-on-a-domain) Abot hands raw pages to an event handler and leaves extraction entirely to the application. The last entry takes a more opinionated approach. ## 7. DotnetSpider: Best for Structured .NET Crawl Pipelines [DotnetSpider](https://github.com/dotnetcore/DotnetSpider) is a higher level crawling framework built around a data flow pipeline. A spider defines its seed requests, a parser turns each response into named values, and a storage step writes the result. The design leans on the .NET generic host, so a spider is configured through `Builder.CreateDefaultBuilder` and dependency injection. Storage targets include console output for development plus database backends for production runs. Three jobs suit DotnetSpider well: - Running entity oriented extraction where every page yields the same named fields. - Scheduling repeated crawls as a hosted service rather than a one off script. - Keeping request scheduling, parsing, and storage as separate, testable stages. The spider below starts on the listing page, follows the pagination links, and prints the product titles from each page. shell```shell dotnet add package DotnetSpider dotnet add package MessagePack ``` csharp```csharp using DotnetSpider; using DotnetSpider.DataFlow; using DotnetSpider.DataFlow.Parser; using DotnetSpider.DataFlow.Storage; using DotnetSpider.Http; using DotnetSpider.Selector; using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; public class ProductSpider( IOptions options, DependenceServices services, ILogger logger) : Spider(options, services, logger) { public static async Task RunAsync() { var builder = Builder.CreateDefaultBuilder(x => x.Speed = 1); await builder.Build().RunAsync(); } protected override async Task InitializeAsync(CancellationToken stoppingToken = default) { AddDataFlow(); AddDataFlow(); await AddRequestsAsync(new Request("https://web-scraping.dev/products")); } private class ProductParser : DataParser { public override Task InitializeAsync() { AddRequiredValidator("web-scraping\\.dev/products"); AddFollowRequestQuerier(Selectors.XPath("//div[@class='paging']")); return Task.CompletedTask; } protected override Task ParseAsync(DataFlowContext context) { var titles = context.Selectable .SelectList(Selectors.XPath("//div[contains(@class, 'product')]//h3/a")) .Select(node => node.Value); context.AddData("Url", context.Request.RequestUri); context.AddData("Products", string.Join(", ", titles)); return Task.CompletedTask; } } } ``` `AddRequiredValidator` restricts the crawl to listing URLs, and `AddFollowRequestQuerier` tells the spider to queue every link inside the pagination block. `ConsoleStorage` prints each parsed record, which keeps the example runnable before a database is wired in. The most recent DotnetSpider release on NuGet is 5.1.7 from July 2024, targeting .NET 8. Confirm the current repository state before adopting the framework for production work. DotnetSpider fetches pages over HTTP rather than through a browser. Client rendered content needs one of the three browser libraries above before extraction can run. [Guide to List Crawling: Everything You Need to KnowComplete list crawling tutorial assess site defenses, bypass anti-bot systems, choose tools (Beautiful Soup, Playwright, Scrapfly), extract data with 6 production-ready code examples, and troubleshoot common failures.](https://scrapfly.io/blog/posts/guide-to-list-crawling) Choose the layer by checking whether the target data exists in the raw response and how many URLs the job must coordinate. ## How Do You Choose Between a C# Parser, Browser, and Crawler? Inspect the raw server response before choosing anything. Where the target data appears in that response, and how many pages the job needs, determine which layer the project actually requires. ### Step 1: Check Whether the Data Is in the Initial HTML Fetch the URL with `HttpClient` and search the returned string for a value visible on the page. When the value is present, a parser is enough, and Html Agility Pack or AngleSharp finishes the job. ### Step 2: Add a Browser Only When Rendering Is Required When the value is missing from the raw response but visible in DevTools, the page builds it after load. Playwright for .NET, Selenium WebDriver, or PuppeteerSharp renders the page and hands back a complete HTML string. Before adding a browser, check whether the page loads that data from a background API request. A direct call to that endpoint is faster and cheaper than rendering the whole page. [How to Scrape Hidden Web DataThe visible HTML doesn't always represent the whole dataset available on the page. In this article, we'll be taking a look at scraping of hidden web data. What is it and how can we scrape it using Python?](https://scrapfly.io/blog/posts/how-to-scrape-hidden-web-data) ### Step 3: Add a Crawler When URL Discovery Becomes the Problem One page needs a parser. A few hundred linked pages need scheduling, deduplication, delay enforcement, and failure handling, which is where Abot or DotnetSpider replaces a hand rolled loop. ### Step 4: Move Retrieval When the Target Blocks the Request A protected target changes the problem again. When responses come back as challenge pages, empty shells, or 403 responses, no parser or crawler on this list fixes the retrieval step. [How to Bypass Anti-Bot Protection When Web ScrapingLearn how anti-bot systems detect scrapers and 5 universal bypass techniques including proxy rotation, fingerprinting, and fortified headless browsers.](https://scrapfly.io/blog/posts/how-to-bypass-anti-bot-protection-when-web-scraping) The first three steps are library choices. The fourth one is an infrastructure choice, and the next section covers what that looks like from a C# project. ## When Should a C# Scraper Use Scrapfly? The parser libraries need an HTML string from another fetch step. The browser and crawler libraries can fetch pages themselves, but proxy orchestration, anti-bot bypass, and distributed execution still belong to the application. A managed [Web Scraping API](https://scrapfly.io/products/web-scraping-api) handles retrieval for targets that block direct HTTP and browser requests. Anti-bot bypass, proxy rotation, and JavaScript rendering run behind one HTTP endpoint that any C# project can call. Scrapfly currently ships official SDKs for Python, TypeScript, Go, and Rust, with no dedicated C# SDK. C# projects call the documented REST endpoint with `HttpClient` instead, which adds no extra dependency. csharp```csharp using System.Text.Json; using AngleSharp.Html.Parser; var apiKey = Environment.GetEnvironmentVariable("SCRAPFLY_API_KEY"); var target = Uri.EscapeDataString("https://web-scraping.dev/product/1"); var endpoint = $"https://api.scrapfly.io/scrape?key={apiKey}&url={target}&unblocker=true&render_js=true"; using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(155) }; var response = await http.GetStringAsync(endpoint); using var payload = JsonDocument.Parse(response); var html = payload.RootElement .GetProperty("result") .GetProperty("content") .GetString(); var document = await new HtmlParser().ParseDocumentAsync(html); var reviews = document.QuerySelectorAll("#reviews .review").Length; Console.WriteLine(reviews); ``` The response arrives as JSON with the rendered page in `result.content`. That string feeds into AngleSharp or Html Agility Pack exactly like the direct `HttpClient` calls used earlier in this guide. Setting `render_js=true` returns the page after client side rendering, so the review elements exist in the returned HTML. Setting `unblocker=true` enables the managed bypass for targets that block ordinary requests. [11 Best Web Scraping APIs, Libraries, and Crawlers for Developers in 2026Compare the best web scraping tools in 2026. Pipeline-based guide covering Scrapfly, BeautifulSoup, Playwright, Scrapy, and more for production scraping.](https://scrapfly.io/blog/posts/best-web-scraping-apis) The FAQ below covers the boundary between parsers, browsers, and crawlers. ## FAQ Is Html Agility Pack or AngleSharp better for C# scraping?Use Html Agility Pack for XPath heavy extraction rules or an existing codebase already built on tolerant parsing. Use AngleSharp for a standards oriented DOM and a CSS selector workflow. Can Html Agility Pack or AngleSharp execute JavaScript?Neither parser executes page JavaScript. When the target data appears only after client side execution, add Playwright for .NET, Selenium WebDriver, or PuppeteerSharp to render the page before parsing. Is Playwright for .NET a web crawler?Playwright controls browser pages and nothing beyond that. A crawler adds URL discovery, scheduling, deduplication, and crawl policies around page retrieval, which is the job Abot and DotnetSpider handle. Which C# library is fastest for web scraping?Speed depends on page size, selector complexity, and whether a browser is involved at all. Benchmark the shortlisted libraries against the pages a project actually needs, in Release configuration, on production hardware. Does Scrapfly have a C# SDK?Scrapfly publishes official SDKs for Python, TypeScript, Go, and Rust, with no dedicated C# package. A C# project calls the documented HTTP API with `HttpClient` and passes the returned HTML to the chosen parser. ## Summary The seven libraries in this guide sit on three layers. Html Agility Pack and AngleSharp parse HTML the application already has. Playwright for .NET, Selenium WebDriver, and PuppeteerSharp produce that HTML for pages a plain request leaves empty. Abot and DotnetSpider handle the layer above both, deciding which URLs get fetched and in what order. Reading the raw server response first is what points a project to the right layer. None of the seven libraries provides a managed anti-bot retrieval layer. When a target answers with challenge pages instead of content, Scrapfly's Web Scraping API handles the bypass and returns clean HTML to the parser. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [Which C# Web Scraping Library Is Best in 2026?](#which-c-web-scraping-library-is-best-in-2026) - [1. Html Agility Pack: Best for XPath Based Static HTML Parsing](#1-html-agility-pack-best-for-xpath-based-static-html-parsing) - [2. AngleSharp: Best for Standards Based DOM and CSS Selectors](#2-anglesharp-best-for-standards-based-dom-and-css-selectors) - [3. Playwright for .NET: Best for Modern JavaScript Applications](#3-playwright-for-net-best-for-modern-javascript-applications) - [4. Selenium WebDriver: Best for Cross Browser WebDriver Workflows](#4-selenium-webdriver-best-for-cross-browser-webdriver-workflows) - [5. PuppeteerSharp: Best for Chromium Focused Automation](#5-puppeteersharp-best-for-chromium-focused-automation) - [6. Abot: Best for Event Driven Site Crawling](#6-abot-best-for-event-driven-site-crawling) - [7. DotnetSpider: Best for Structured .NET Crawl Pipelines](#7-dotnetspider-best-for-structured-net-crawl-pipelines) - [How Do You Choose Between a C# Parser, Browser, and Crawler?](#how-do-you-choose-between-a-c-parser-browser-and-crawler) - [Step 1: Check Whether the Data Is in the Initial HTML](#step-1-check-whether-the-data-is-in-the-initial-html) - [Step 2: Add a Browser Only When Rendering Is Required](#step-2-add-a-browser-only-when-rendering-is-required) - [Step 3: Add a Crawler When URL Discovery Becomes the Problem](#step-3-add-a-crawler-when-url-discovery-becomes-the-problem) - [Step 4: Move Retrieval When the Target Blocks the Request](#step-4-move-retrieval-when-the-target-blocks-the-request) - [When Should a C# Scraper Use Scrapfly?](#when-should-a-c-scraper-use-scrapfly) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ tools data-parsing ### HtmlAgilityPack vs AngleSharp: Which C# HTML Parser Should You Use? A fixture based comparison of Html Agility Pack and AngleSharp for C# HTML parsing, covering XPath versus CSS selectors,... ](https://scrapfly.io/blog/posts/htmlagilitypack-vs-anglesharp) [ python headless-browser ### Web Scraping with Selenium and Python Introduction to web scraping dynamic javascript powered websites and web apps using Selenium browser automation library ... ](https://scrapfly.io/blog/posts/web-scraping-with-selenium-and-python) [ python selenium ### Intro to Web Scraping Using Selenium Grid In this guide, you will learn about installing and configuring Selenium Grid with Docker and how to use it for web scrap... ](https://scrapfly.io/blog/posts/intro-to-web-scraping-using-selenium-grid) ## Related Questions - [ Q How to scroll to the bottom of the page with Playwright? ](https://scrapfly.io/blog/answers/how-to-scroll-to-the-bottom-with-playwright) Extract structured data with AI, **1,000 free credits** [Start Free](https://scrapfly.io/register)