[Blog](https://scrapfly.io/blog) / [data-parsing](https://scrapfly.io/blog/tag/data-parsing) / [Web Scraping with Go: Colly, goquery, and Browser Tools in 2026](https://scrapfly.io/blog/posts/web-scraping-with-go) # Web Scraping with Go: Colly, goquery, and Browser Tools in 2026 by [Mazen Ramadan](https://scrapfly.io/blog/author/mazen) Aug 24, 2026 33 min read [\#data-parsing](https://scrapfly.io/blog/tag/data-parsing) [\#frameworks](https://scrapfly.io/blog/tag/frameworks) [\#golang](https://scrapfly.io/blog/tag/golang) [\#http](https://scrapfly.io/blog/tag/http) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-go "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-go&text=Web%20Scraping%20with%20Go%3A%20Colly%2C%20goquery%2C%20and%20Browser%20Tools%20in%202026 "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-go "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-go) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-go) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-go) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-go) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-go) Go's standard library ships an HTTP client and nothing else. The first real decision in a Go scraping project is which library handles HTML parsing, crawling, and JavaScript rendering. Picking wrong usually means rewriting the whole thing later. This guide answers the library question first, then builds a scraper end to end. It covers net/http and goquery for a single page, Colly for a full crawl, and a browser tool for pages that need JavaScript. Every code block runs against a live target with checked errors, ready to copy into a project. ## Key Takeaways - **net/http and goquery fit small, static jobs**, where you own retries and rate limits. - **Colly crawls, it doesn't render**, adding concurrency, per-domain limits, and caching. - **Rod, chromedp, and Playwright-Go add real JavaScript rendering** to Go scrapers. - **Ferret moves extraction rules into a query language**, outside your Go code. - **goquery only parses, and Colly only crawls**, so match the tool to the job. - **Protected or high-scale targets need a managed layer**, whatever library you pick. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## Which Go Web Scraping Library Should You Use? Use net/http with [goquery](https://github.com/PuerkitoBio/goquery) for small static jobs. Reach for [Colly](https://github.com/gocolly/colly) when you're crawling many pages and need concurrency and rate limiting built in. Move to a browser library when the data you need isn't in the raw HTML. [Rod](https://github.com/go-rod/rod) and [chromedp](https://github.com/chromedp/chromedp)both fit that case. Pick Ferret when extraction rules should live outside your Go code. | Option | Category | JavaScript | Crawl controls | Best for | Main trade-off | |---|---|---|---|---|---| | net/http + goquery | Fetch + parse | No | No | Small static jobs, maximum control | You own retries, queues, rate limits | | htmlquery | Parse (XPath) | No | No | XPath-shaped selectors on static HTML | Parser only, pairs with net/http | | Colly | Crawling framework | No | Yes | Concurrent static crawls at scale | Callback architecture, no rendering | | Rod | Browser control | Yes (Chromium) | No | JS-rendered sites, fluent Go API | Browser cost, stealth is a separate package | | chromedp | Browser control (CDP) | Yes (Chromium) | No | Lower-level Chrome automation | More orchestration work | | Playwright-Go | Browser binding | Yes (3 engines) | No | Cross-browser requirements | Community-maintained, heavier setup | | Ferret | Declarative extraction | Via HTTP or CDP driver | Workflow-dependent | Extraction rules outside Go code | Smaller ecosystem, FQL learning curve | What these tools do: - goquery is a parser, not a scraper. It doesn't fetch anything. You hand it a response body. - Colly is a crawler framework. It orchestrates fetching, queueing, rate limiting, caching, and cookies, and it uses goquery internally for selection. - Browser controllers aren't crawler frameworks. Rod, chromedp, and [Playwright-Go](https://github.com/mxschmitt/playwright-go) render pages, and you build the queue, the per-domain limits, and the cache around them yourself. - Ferret moves extraction rules out of your Go code into a query language, which is a team and workflow choice more than a performance one. Is the data in the raw HTML? Check with `curl` or view-source before reaching for a browser. One page or many? A handful of pages calls for net/http and goquery. Many pages, needing concurrency and politeness, call for Colly. Not in the raw HTML? Look for a JSON or XHR endpoint first, since it's usually cheaper than a browser. Reach for Rod or chromedp, or Playwright-Go if you need Firefox or WebKit. Playwright-Go is community-maintained, not an official Microsoft port. And Rod's stealth features live in a separate package that doesn't guarantee it bypasses anything. ### Static HTML: net/http, goquery, and htmlquery For a page that already contains the data in its HTML, net/http fetches it. goquery or [htmlquery](https://github.com/antchfx/htmlquery) parses it. The Setup and Core Concepts sections below build this path first, since it's the foundation every other approach in this guide builds on. ### Crawling Frameworks: Colly Once a job means following links across many pages, a bare net/http loop turns into heavy manual bookkeeping for concurrency and rate limits. The Web Scraping With Colly section covers the framework that handles that bookkeeping for you. ### JavaScript Rendering: Rod, chromedp, and Playwright-Go When the data you need only shows up after JavaScript runs, none of the above helps. The How Do You Scrape JavaScript Websites in Go section compares Rod, chromedp, and Playwright-Go, then walks through a rendered example with chromedp. ### Declarative Extraction: Ferret [Ferret](https://github.com/MontFerret/ferret) runs extraction logic written in its own query language instead of Go code. That suits teams that want scraping rules to live outside a compiled binary rather than inside it. ## Setup: Go Environment and Packages Building a Go scraper needs a working Go install and three third-party packages: goquery, htmlquery, and Colly. ### Setting Up the Go Environment Download the Go release for your platform from [the official Go downloads page](https://go.dev/dl/). Then follow the [installation guide](https://go.dev/doc/install) to add Go to your `PATH`. Confirm the install with: bash```bash go version go version go1.27.0 linux/amd64 ``` The exact version string depends on your OS and architecture, but the command confirms Go is on your `PATH` and ready to build the examples below. ### Installing the Packages Create a new module, then pull in goquery for CSS selectors, htmlquery for XPath, and Colly for the crawling framework: bash```bash go mod init product-scraper go get github.com/PuerkitoBio/goquery go get github.com/antchfx/htmlquery go get github.com/gocolly/colly/v2 ``` Each `go get` records the resolved version in `go.mod` and `go.sum`. Tested with Go 1.27.0, goquery v1.12.0, htmlquery v1.3.6, and Colly v2.3.0 on 2026-08-24. ## Web Scraping With Go: Core Concepts Every scraper built for this guide checks its errors. A scraper that ignores a failed request and then reads its response will panic instead of failing with a clear message. ### Sending HTTP Requests A request goes out, and the server's response carries the data you're after. Here's a minimal, complete request against a real target, with a reusable client, a timeout, and every error checked before it's used: golang```golang package main import ( "context" "fmt" "io" "net/http" "os" "time" ) var client = &http.Client{Timeout: 10 * time.Second} func check(err error) { if err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } } func main() { req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, "https://web-scraping.dev/product/1", nil) check(err) resp, err := client.Do(req) check(err) defer resp.Body.Close() if resp.StatusCode < 200 || resp.StatusCode >= 300 { check(fmt.Errorf("unexpected status: %d", resp.StatusCode)) } body, err := io.ReadAll(resp.Body) check(err) fmt.Printf("status: %d\n", resp.StatusCode) fmt.Printf("content-type: %s\n", resp.Header.Get("Content-Type")) fmt.Printf("body bytes: %d\n", len(body)) } ``` text```text status: 200 content-type: text/html; charset=utf-8 body bytes: 33767 ``` The shared `client` and its `check()` helper get reused for every request path, so building the request, sending it, and reading the body all fail loudly instead of silently. The status check stops the scraper before a 404 or 500 page reaches a parser. Body size varies slightly between requests because the page serves some content that changes, so treat the byte count as a sanity check rather than a fixed value. #### Request Methods, Headers, and Body Changing the HTTP method only means passing a different value to `NewRequestWithContext`: | Method | Typical use in scraping | |---|---| | `GET` | Fetch a page or API response | | `POST` | Submit a form or send a JSON payload | | `HEAD` | Check status or headers without a body | Set headers on the request before sending it: golang```golang req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36") req.Header.Set("Cookie", "cookie_key=cookie_value;") ``` Above, the request carries a [How to Handle Cookies in Web Scraping](https://scrapfly.io/blog/posts/how-to-handle-cookies-in-web-scraping) header. It also carries a [How to Effectively Use User Agents for Web Scraping](https://scrapfly.io/blog/posts/user-agent-header-in-web-scraping) header that matches a real browser. Don't set `Accept-Encoding` manually. Go's transport only decompresses a response automatically when it added that header itself. A hand-set `Accept-Encoding: gzip` hands a parser compressed bytes it can't read. For a POST with a body, pass an `io.Reader` instead of `nil` and set `Content-Type`: golang```golang payload := "page=1&test=2&foo=bar" req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, strings.NewReader(payload)) check(err) req.Header.Set("Content-Type", "application/x-www-form-urlencoded") ``` Both fragments extend the client and `check()` helper from the request above. Only the method, headers, or body change. Headers matter beyond compression, since sites and anti-bot systems use them to spot automated requests. Our [guide to avoiding blocks with headers](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-headers) covers that in more depth. ### Parsing HTML With goquery (CSS Selectors) CSS selectors are the most common way to pick elements out of an HTML document, and goquery brings jQuery-style selection to Go. Here's a function that requests a product page and pulls out its title, price, description, and reviews: golang```golang package main import ( "encoding/json" "fmt" "log" "net/http" "strconv" "strings" "github.com/PuerkitoBio/goquery" ) type Review struct { Date string `json:"date"` Rating int `json:"rating"` Text string `json:"text"` } type Product struct { Title string Price float64 Description string Reviews []Review } func parseProduct(url string) (Product, error) { var product Product resp, err := http.Get(url) if err != nil { return product, fmt.Errorf("request failed: %w", err) } defer resp.Body.Close() if resp.StatusCode != http.StatusOK { return product, fmt.Errorf("unexpected status: %d", resp.StatusCode) } doc, err := goquery.NewDocumentFromReader(resp.Body) if err != nil { return product, fmt.Errorf("failed to parse HTML: %w", err) } product.Title = doc.Find("h3.card-title").Text() priceParts := strings.SplitN(doc.Find("span.product-price").Text(), "$", 2) if len(priceParts) == 2 { price, err := strconv.ParseFloat(strings.TrimSpace(priceParts[1]), 64) if err != nil { return product, fmt.Errorf("failed to parse price: %w", err) } product.Price = price } product.Description = doc.Find("p.product-description").Text() reviewsJSON := doc.Find("script#reviews-data").Text() if err := json.Unmarshal([]byte(reviewsJSON), &product.Reviews); err != nil { return product, fmt.Errorf("failed to parse reviews: %w", err) } return product, nil } func main() { product, err := parseProduct("https://web-scraping.dev/product/1") if err != nil { log.Fatal(err) } fmt.Printf("Title: %s\n", product.Title) fmt.Printf("Price: $%.2f\n", product.Price) fmt.Printf("Description: %.80s...\n", product.Description) fmt.Printf("Reviews: %d\n", len(product.Reviews)) } ``` text```text Title: Box of Chocolate Candy Price: $9.99 Description: Indulge your sweet tooth with our Box of Chocolate Candy. Each box contains an a... Reviews: 5 ``` `doc.Find()` locates elements by their CSS selector. Every fallible step (the request, the parse, the price conversion, the JSON decode) returns an error instead of an empty or wrong value. The caller has to handle each one. Note the selector is `span.product-price`, not a bare `.description`. That bare class doesn't exist anywhere in this page's markup. For more on selecting elements this way, see our guide to [Parsing HTML with CSS SelectorsIntroduction to using CSS selectors to parse web-scraped content. Best practices, available tools and common challenges by interactive examples.](https://scrapfly.io/blog/posts/parsing-html-with-css) ### Parsing HTML With htmlquery (XPath) XPath is a more powerful alternative to CSS selectors, useful when a selector needs to match on text content or move up the tree. Here's the same product page parsed with htmlquery: golang```golang package main import ( "fmt" "log" "strconv" "strings" "github.com/antchfx/htmlquery" ) type Product struct { Name string Price float64 Description string Image string } func main() { doc, err := htmlquery.LoadURL("https://web-scraping.dev/product/1") if err != nil { log.Fatalf("failed to load page: %v", err) } nameNode := htmlquery.FindOne(doc, "//h3[contains(@class,'card-title')]") priceNode := htmlquery.FindOne(doc, "//span[contains(@class,'product-price') and not(contains(@class,'product-price-full'))]") priceText := strings.TrimSpace(strings.TrimPrefix(htmlquery.InnerText(priceNode), "$")) price, err := strconv.ParseFloat(priceText, 64) if err != nil { log.Fatalf("failed to parse price %q: %v", priceText, err) } descNode := htmlquery.FindOne(doc, "//p[contains(@class,'product-description')]") imgNode := htmlquery.FindOne(doc, "//img[contains(@class,'product-img')]") product := Product{ Name: strings.TrimSpace(htmlquery.InnerText(nameNode)), Price: price, Description: strings.TrimSpace(htmlquery.InnerText(descNode)), Image: htmlquery.SelectAttr(imgNode, "src"), } fmt.Printf("Name: %s\n", product.Name) fmt.Printf("Price: $%.2f\n", product.Price) fmt.Printf("Description: %.80s...\n", product.Description) fmt.Printf("Image: %s\n", product.Image) } ``` text```text Name: Box of Chocolate Candy Price: $9.99 Description: Indulge your sweet tooth with our Box of Chocolate Candy. Each box contains an a... Image: https://web-scraping.dev/assets/products/orange-chocolate-box-small-1.webp ``` The price selector needs `not(contains(@class,'product-price-full'))`, because XPath's `contains()` does substring matching. Without that guard it also matches a sibling element whose class includes `product-price-full`. `InnerText()` reads text content and `SelectAttr()` reads an attribute, the XPath equivalents of goquery's `.Text()` and `.Attr()`. For more on XPath syntax, see our guide to [Parsing HTML with XpathIntroduction to xpath in the context of web-scraping. How to extract data from HTML documents using xpath, best practices and available tools.](https://scrapfly.io/blog/posts/parsing-html-with-xpath) ### Crawling Links Between Pages Crawling means following the links inside a page instead of stopping at the first response. The tricky part in Go is that `href` values aren't always absolute. A link like `/products?page=2` needs resolving against the page's own URL before it's usable. Here's a reusable function that does that: golang```golang func discoverLinks(doc *goquery.Document, selector string) ([]string, error) { if doc.Url == nil { return nil, errors.New("document has no base URL set") } var links []string doc.Find(selector).Each(func(_ int, s *goquery.Selection) { href, ok := s.Attr("href") if ref, err := url.Parse(href); ok && err == nil { links = append(links, doc.Url.ResolveReference(ref).String()) } }) return links, nil } ``` `goquery.NewDocumentFromReader` doesn't set `doc.Url` on its own, so every call site has to set `doc.Url = resp.Request.URL` right after parsing, or `discoverLinks` returns its base-URL error. Here's the full, runnable version against a real page: golang```golang package main import ( "errors" "fmt" "net/http" "net/url" "time" "github.com/PuerkitoBio/goquery" ) func discoverLinks(doc *goquery.Document, selector string) ([]string, error) { if doc.Url == nil { return nil, errors.New("document has no base URL set") } var links []string doc.Find(selector).Each(func(_ int, s *goquery.Selection) { href, ok := s.Attr("href") if ref, err := url.Parse(href); ok && err == nil { links = append(links, doc.Url.ResolveReference(ref).String()) } }) return links, nil } func main() { client := &http.Client{Timeout: 15 * time.Second} resp, err := client.Get("https://web-scraping.dev/products") if err != nil { panic(err) } defer resp.Body.Close() if resp.StatusCode != http.StatusOK { panic(fmt.Sprintf("unexpected status: %d", resp.StatusCode)) } doc, err := goquery.NewDocumentFromReader(resp.Body) if err != nil { panic(err) } doc.Url = resp.Request.URL fmt.Println("== product links (already-absolute hrefs, selector: div.product h3 a) ==") productLinks, err := discoverLinks(doc, "div.product h3 a") if err != nil { panic(err) } for _, l := range productLinks { fmt.Println(l) } fmt.Printf("count: %d\n\n", len(productLinks)) fmt.Println("== nav links (relative hrefs, selector: nav a) ==") navLinks, err := discoverLinks(doc, "nav a") if err != nil { panic(err) } for _, l := range navLinks { fmt.Println(l) } fmt.Printf("count: %d\n", len(navLinks)) } ``` text```text == product links (already-absolute hrefs, selector: div.product h3 a) == https://web-scraping.dev/product/1 https://web-scraping.dev/product/2 https://web-scraping.dev/product/3 https://web-scraping.dev/product/4 https://web-scraping.dev/product/5 count: 5 == nav links (relative hrefs, selector: nav a) == https://web-scraping.dev/ https://web-scraping.dev/products https://web-scraping.dev/docs https://web-scraping.dev/api/graphql https://web-scraping.dev/products https://web-scraping.dev/reviews https://web-scraping.dev/testimonials https://web-scraping.dev/file-download https://web-scraping.dev/mcp-tools https://web-scraping.dev/login https://web-scraping.dev/cart count: 11 ``` The product links come back already absolute on this page, so resolving them is a no-op. The nav links are root-relative, and `ResolveReference` turns both cases into full URLs without branching on which one it got. The full example below reuses this exact function. For crawling theory beyond Go specifics, see our guide to [crawling with Python](https://scrapfly.io/blog/posts/crawling-with-python). For the distinction this section relies on, see [what separates scraping from crawling](https://scrapfly.io/blog/answers/whats-the-difference-between-scraping-and-crawling). ## Example Go Scraper: net/http and goquery Here's a full scraper that follows pagination across `web-scraping.dev/products`, pulls every product's details and reviews, and saves the results to JSON: golang```golang package main import ( "encoding/json" "fmt" "log" "net/http" "os" "regexp" "strconv" "github.com/PuerkitoBio/goquery" ) type Review struct { Date string Rating int Text string } type Product struct { Name string Price float64 Currency string Image string Description string Link string Reviews []Review } var totalPagesRe = regexp.MustCompile(`(\d+) pages`) // requestPage sends a GET request with basic browser-like headers and // returns an error instead of silently swallowing one. func requestPage(url string) (*http.Response, error) { client := &http.Client{} req, err := http.NewRequest("GET", url, nil) if err != nil { return nil, fmt.Errorf("building request for %s: %w", url, err) } req.Header.Set("Accept", "text/html") req.Header.Set("Accept-Language", "en-US,en;q=0.9") req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:154.0) Gecko/20100101 Firefox/154.0") resp, err := client.Do(req) if err != nil { return nil, fmt.Errorf("requesting %s: %w", url, err) } return resp, nil } // crawlReviews fetches the reviews embedded in a product page's hidden // script tag. func crawlReviews(url string) ([]Review, error) { resp, err := requestPage(url) if err != nil { return nil, err } defer resp.Body.Close() doc, err := goquery.NewDocumentFromReader(resp.Body) if err != nil { return nil, fmt.Errorf("parsing reviews page %s: %w", url, err) } reviewsScript := doc.Find("script#reviews-data").Text() var reviews []Review if err := json.Unmarshal([]byte(reviewsScript), &reviews); err != nil { return nil, fmt.Errorf("decoding reviews JSON for %s: %w", url, err) } return reviews, nil } // parseProducts reads every product card on a listing page. func parseProducts(doc *goquery.Document) []Product { var products []Product doc.Find("div.row.product").Each(func(i int, sel *goquery.Selection) { nameLink := sel.Find("h3.mb-0 > a") link, _ := nameLink.Attr("href") // ok if missing: parseFloat below still logs the row price, err := strconv.ParseFloat(sel.Find("div.price").Text(), 64) if err != nil { log.Printf("could not parse price for %s: %v", link, err) } reviews, err := crawlReviews(link) if err != nil { log.Printf("could not fetch reviews for %s: %v", link, err) } products = append(products, Product{ Name: nameLink.Text(), Price: price, Currency: "$", Image: sel.Find("img").AttrOr("src", ""), Description: sel.Find("div.short-description").Text(), Link: link, Reviews: reviews, }) }) return products } // totalPages reads the "N pages" hint from the listing page. It's used only // to log progress, never to drive the crawl loop, so a miss here doesn't // stop the scrape. func totalPages(doc *goquery.Document) (int, error) { pagingStr := doc.Find(".paging-meta").Text() match := totalPagesRe.FindStringSubmatch(pagingStr) if match == nil { return 0, fmt.Errorf("no page count found in %q", pagingStr) } return strconv.Atoi(match[1]) } // nextPageURL reads the trailing ">" link inside div.paging. On the last // page that anchor has no href attribute at all, which is the signal to // stop crawling instead of requesting an empty URL. func nextPageURL(doc *goquery.Document) (string, bool) { href, exists := doc.Find("div.paging a").Last().Attr("href") if !exists || href == "" { return "", false } return href, true } // scrapeProducts follows the "next" link in div.paging until the site stops // offering one, collecting every product along the way. func scrapeProducts(startURL string) []Product { var allProducts []Product nextURL := startURL loggedTotal := false for nextURL != "" { resp, err := requestPage(nextURL) if err != nil { log.Printf("stopping crawl, request failed: %v", err) break } log.Printf("scraping page: %s", nextURL) doc, err := goquery.NewDocumentFromReader(resp.Body) resp.Body.Close() if err != nil { log.Printf("stopping crawl, could not parse %s: %v", nextURL, err) break } if !loggedTotal { if n, err := totalPages(doc); err != nil { log.Printf("could not determine total page count: %v", err) } else { log.Printf("found %d pages to crawl", n) } loggedTotal = true } allProducts = append(allProducts, parseProducts(doc)...) next, ok := nextPageURL(doc) if !ok { break } nextURL = next } return allProducts } // saveToJSON writes the scraped products to a JSON file, surfacing every // error instead of ignoring it. func saveToJSON(products []Product, fileName string) error { jsonData, err := json.MarshalIndent(products, "", " ") if err != nil { return fmt.Errorf("encoding products: %w", err) } file, err := os.Create(fileName + ".json") if err != nil { return fmt.Errorf("creating %s.json: %w", fileName, err) } defer file.Close() if _, err := file.Write(jsonData); err != nil { return fmt.Errorf("writing %s.json: %w", fileName, err) } fmt.Printf("Saved %d products to %s.json\n", len(products), fileName) return nil } func main() { products := scrapeProducts("https://web-scraping.dev/products") if err := saveToJSON(products, "product_data"); err != nil { log.Fatalf("could not save products: %v", err) } } ``` text```text 2026/08/17 21:06:06 scraping page: https://web-scraping.dev/products 2026/08/17 21:06:06 found 6 pages to crawl 2026/08/17 21:06:08 scraping page: https://web-scraping.dev/products?page=2 2026/08/17 21:06:09 scraping page: https://web-scraping.dev/products?page=3 2026/08/17 21:06:10 scraping page: https://web-scraping.dev/products?page=4 2026/08/17 21:06:11 scraping page: https://web-scraping.dev/products?page=5 Saved 25 products to product_data.json ``` `requestPage` sets basic browser-like headers, deliberately without a manual `Accept-Encoding`, and returns an error instead of a bare response. `nextPageURL` stops the loop the moment the pager's trailing link has no `href`. That happens one page before the site's own `.paging-meta` text claims the crawl finishes. This fixture's pager links only ever reach 5 pages and 25 products, even though its own text says 6 pages and 28 results. The crawl above covers every page the site's pager links to, not the number in that label. Links here came back already absolute, but production code should still resolve them with the `discoverLinks` pattern above, since not every target's markup will. The scraper above uses only the standard library and two parsing packages. A crawling framework takes over the bookkeeping from here. ## Web Scraping With Colly Colly is a dedicated crawling framework for Go, built on top of goquery for HTML selection. It handles a lot of the bookkeeping a hand-rolled crawler would otherwise need: - Built-in caching middleware - Asynchronous, synchronous, and parallel execution - Distributed scraping, request delays, and maximum concurrency - Automatic cookie and session handling Colly's own project README describes it as fast, citing more than 1,000 requests per second on a single core. That's the project's own claim, not a number this guide measured. ### How Colly Works: Collectors A Colly scraper needs at least one collector, the component that manages requests, responses, and callbacks. Here's one configured with a custom User-Agent, a depth limit, an allowed domain, and async execution, calling `Wait()` before the program exits: golang```golang package main import ( "fmt" "log" "github.com/gocolly/colly/v2" ) func main() { c := colly.NewCollector( colly.UserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"), colly.MaxDepth(1), colly.AllowedDomains("web-scraping.dev"), colly.Async(true), ) c.OnRequest(func(r *colly.Request) { fmt.Println("visiting", r.URL) }) c.OnResponse(func(r *colly.Response) { fmt.Println("status", r.StatusCode, "bytes", len(r.Body)) }) c.OnError(func(r *colly.Response, err error) { log.Println("request failed:", err) }) if err := c.Visit("https://web-scraping.dev/product/1"); err != nil { log.Fatal("visit failed:", err) } c.Wait() } ``` text```text visiting https://web-scraping.dev/product/1 status 200 bytes ~33770 ``` `colly.Async(true)` runs requests on a goroutine, so `Wait()` has to block until it finishes, or the program exits before the request completes. The byte count above varies a little between runs, the same pattern as the net/http example earlier. You can also configure a collector with these options, verified against Colly v2: | Option | Description | |---|---| | `UserAgent` | Sets the User-Agent header on every request | | `MaxDepth` | Limits recursion depth when crawling, 0 means unlimited | | `AllowedDomains` | Restricts requests to a list of domains | | `DisallowedDomains` | Blocks a list of domains | | `AllowURLRevisit` | Allows requesting the same URL more than once | | `MaxBodySize` | Caps response body size in bytes, 0 means unlimited | | `CacheDir` | Caches GET responses to disk, disabled if unset | | `IgnoreRobotsTxt` | Ignores the target's robots.txt rules | | `Async` | Runs requests on goroutines instead of blocking | For the full option list, see the [Colly v2 API reference](https://pkg.go.dev/github.com/gocolly/colly/v2#Collector). ### How Colly Works: Callbacks Callbacks are functions Colly runs at specific points in a request's lifecycle: | Callback | Fires | Typical use | |---|---|---| | `OnRequest` | Before sending the request | Logging, setting headers | | `OnHTML` | For each matching HTML element | Extraction, following links | | `OnXML` | For each matching XML/XPath node | Sitemaps, feeds | | `OnError` | On request failure | Retries, logging | | `OnResponse` | After the response arrives | Raw body handling | | `OnScraped` | After all other callbacks finish | Flush, aggregate, finalize | `OnRequest` and `OnHTML` together handle logging and extraction: golang```golang collector.OnRequest(func(r *colly.Request) { log.Println("visiting", r.URL) }) collector.OnHTML("div.row.product", func(e *colly.HTMLElement) { name := e.ChildText("h3 a") price := e.ChildText("div.price") log.Println("product:", name, price) }) ``` text```text 2026/08/17 21:02:25 visiting https://web-scraping.dev/products 2026/08/17 21:02:26 product: Box of Chocolate Candy 24.99 2026/08/17 21:02:26 product: Dark Red Energy Potion 4.99 2026/08/17 21:02:26 product: Teal Energy Potion 4.99 2026/08/17 21:02:26 product: Red Energy Potion 4.99 2026/08/17 21:02:26 product: Blue Energy Potion 4.99 ``` `OnError` and `OnScraped` together handle failures and cleanup: golang```golang collector.OnError(func(r *colly.Response, err error) { log.Println("request failed:", r.Request.URL, "-", err) }) collector.OnScraped(func(r *colly.Response) { log.Println("finished scraping", r.Request.URL) }) ``` text```text 2026/08/17 21:02:26 finished scraping https://web-scraping.dev/products 2026/08/17 21:02:26 visiting https://web-scraping.dev/does-not-exist-page 2026/08/17 21:02:26 request failed: https://web-scraping.dev/does-not-exist-page - Not Found ``` Both blocks ran as one program against `collector`, a single `colly.NewCollector()`. Note that `OnScraped` doesn't fire on the failed visit, only `OnRequest` and `OnError` do, since there's nothing left to scrape once the request itself failed. ### Example Colly Scraper This crawler uses two collectors: `searchCollector` paginates the listing pages, and `productCollector` requests each product page it finds. Both collectors log their own errors, and a mutex guards the shared `products` slice, since `Async(true)` runs callbacks on multiple goroutines: golang```golang package main import ( "encoding/json" "fmt" "log" "strconv" "strings" "sync" "github.com/gocolly/colly/v2" ) type Review struct { Date string Rating int Text string } type Product struct { Name string Price float64 Currency string Image string Description string Link string Reviews []Review } func main() { c := colly.NewCollector( colly.CacheDir("./web-scraping.dev_cache"), // Cache responses to prevent multiple downloads colly.Async(true), // Enable asynchronous execution ) // Clone the created collector into a pagination collector searchCollector := c.Clone() // Clone it again into a product collector productCollector := c.Clone() var products []Product var mu sync.Mutex // protects products across concurrent OnHTML callbacks // Paginate search pages searchCollector.OnHTML("div.paging", func(e *colly.HTMLElement) { links := e.ChildAttrs("a", "href") if len(links) == 0 { // No links carry an href on this page at all return } nextPage := links[len(links)-1] if strings.Contains(nextPage, "https://web-scraping.dev") { e.Request.Visit(nextPage) } }) // Log before visiting each search page searchCollector.OnRequest(func(r *colly.Request) { log.Println("Visiting search page", r.URL.String()) }) // Log failed search page requests instead of losing them silently searchCollector.OnError(func(r *colly.Response, err error) { log.Println("search request failed", r.Request.URL, err) }) // On every product link, queue the product details page searchCollector.OnHTML("div.row.product div h3 a", func(e *colly.HTMLElement) { productLink := e.Attr("href") productCollector.Visit(productLink) }) // Log before visiting each product page productCollector.OnRequest(func(r *colly.Request) { log.Println("Visiting product page", r.URL.String()) }) // Log failed product page requests instead of losing them silently productCollector.OnError(func(r *colly.Response, err error) { log.Println("product request failed", r.Request.URL, err) }) // Parse each product page productCollector.OnHTML("body", func(e *colly.HTMLElement) { var reviews []Review reviewsScript := e.ChildText("script#reviews-data") if err := json.Unmarshal([]byte(reviewsScript), &reviews); err != nil { log.Println("could not parse reviews for", e.Request.URL, err) } priceParts := strings.Split(e.ChildText("span.product-price"), "$") if len(priceParts) < 2 { log.Println("no price found on", e.Request.URL) return } price, err := strconv.ParseFloat(strings.TrimSpace(priceParts[1]), 64) if err != nil { log.Println("could not parse price on", e.Request.URL, err) return } product := Product{ Name: e.ChildText("h3.card-title"), Price: price, Currency: "$", Image: e.ChildAttr("img", "src"), Description: e.ChildText("p.product-description"), Link: e.Request.URL.String(), Reviews: reviews, } mu.Lock() products = append(products, product) mu.Unlock() }) // Start scraping from the main products page searchCollector.Visit("https://web-scraping.dev/products") // Wait for both collectors to finish searchCollector.Wait() productCollector.Wait() fmt.Println("scraped", len(products), "products") for _, p := range products { fmt.Printf("%s - $%.2f - %d reviews\n", p.Name, p.Price, len(p.Reviews)) } } ``` text```text scraped 25 products Blue Energy Potion - $4.99 - 5 reviews Box of Chocolate Candy - $9.99 - 5 reviews Teal Energy Potion - $4.99 - 4 reviews Dark Red Energy Potion - $4.99 - 4 reviews Red Energy Potion - $4.99 - 4 reviews Dragon Energy Potion - $4.99 - 4 reviews Running Shoes for Men - $49.99 - 5 reviews Kids' Light-Up Sneakers - $29.99 - 5 reviews Hiking Boots for Outdoor Adventures - $89.99 - 5 reviews Women's High Heel Sandals - $59.99 - 5 reviews Classic Leather Sneakers - $79.99 - 5 reviews ... (14 more lines omitted) ``` The full run prints all 25 lines. The block above shows a trimmed excerpt, and `Async(true)`means the goroutine order shifts between runs, so this exact sequence won't repeat. `go build -race` and `go run -race` on this exact program reported zero data races. Like the net/http example above, this fixture's own pager links only reach 5 pages and 25 products. ### Rate Limits, Retries, and Error Handling Colly gives you the controls for pacing and retrying requests, but none of them are on by default. `LimitRule` sets a delay and a parallelism cap per domain, and `OnError` is where a bounded retry belongs: golang```golang c.AllowURLRevisit = true // without this, c.Visit(target) below silently no-ops on retry c.Limit(&colly.LimitRule{ DomainGlob: "*web-scraping.dev*", Parallelism: 2, Delay: 500 * time.Millisecond, RandomDelay: 500 * time.Millisecond, }) retries := make(map[string]int) var retriesMu sync.Mutex const maxRetries = 3 c.OnError(func(r *colly.Response, err error) { target := r.Request.URL.String() retriesMu.Lock() retries[target]++ attempt := retries[target] retriesMu.Unlock() if attempt <= maxRetries { log.Printf("retry %d/%d for %s: %v", attempt, maxRetries, target, err) c.Visit(target) return } log.Printf("giving up on %s after %d retries: %v", target, maxRetries, err) }) ``` `DomainGlob` scopes the rule to a domain pattern, `Parallelism` caps concurrent requests, and `Delay` plus `RandomDelay` space them out so a burst doesn't look automated. Colly refuses to visit the same URL twice by default, so a retry loop needs `AllowURLRevisit` set. Otherwise the `c.Visit(target)` call inside `OnError` does nothing, and the retry counter increments without ever firing a second request. The counter tracks attempts per URL and gives up after `maxRetries`, instead of retrying forever or dropping a failed page silently. Sites that return [HTTP 429](https://scrapfly.io/blog/posts/what-is-http-error-429-too-many-requests) responses are exactly the case this pattern exists for. ### Using Proxies With Colly Rotating IP addresses spreads a crawl's traffic across multiple addresses instead of hammering a target from one. Colly's `proxy` package includes a round-robin switcher for exactly that: golang```golang p, err := proxy.RoundRobinProxySwitcher( "socks5://some_proxy_domain:1234", "http://some_proxy_domain:1234", ) if err != nil { log.Fatal(err) } c.SetProxyFunc(p) ``` [Best Proxy Providers for Web Scraping: Choose by Type (2026)Compare datacenter, residential, ISP, and mobile proxies by decision gate, then choose direct inventory, managed fetching, or an egress optimizer.](https://scrapfly.io/blog/posts/best-proxy-providers-for-web-scraping) For custom switching logic, parse each proxy URL with `url.Parse` instead of building a `url.URL` by hand. Putting a scheme string into the `Host` field, as in `&url.URL{Host: "socks5://..."}`, produces a URL that won't route: golang```golang var proxyURLs = []string{ "socks5://some_proxy_domain:1234", "http://some_proxy_domain:1234", } func randomProxySwitcher(_ *http.Request) (*url.URL, error) { raw := proxyURLs[rand.Intn(len(proxyURLs))] return url.Parse(raw) } c.SetProxyFunc(randomProxySwitcher) ``` `url.Parse` splits the scheme, host, and port correctly, so the returned `*url.URL` routes through the intended proxy. For more on proxy rotation, see our guide to [using proxies in web scraping](https://scrapfly.io/blog/posts/introduction-to-proxies-in-web-scraping). Our guide to [rotating proxies](https://scrapfly.io/blog/posts/how-to-rotate-proxies-in-web-scraping) covers rotation strategy in more depth. ### Concurrency and Safe Result Collection Colly runs `OnHTML` callbacks on multiple goroutines once you set `Async(true)`, so appending to a shared slice from inside a callback, without synchronization, is a data race. Two goroutines can write to the same slice at once and corrupt it. The two-collector example above already applies the fix, a `sync.Mutex` guarding every append to the shared `products` slice: golang```golang var products []Product var mu sync.Mutex productCollector.OnHTML("body", func(e *colly.HTMLElement) { product := Product{ /* ... fields from the parsed page ... */ } mu.Lock() products = append(products, product) mu.Unlock() }) ``` Running that exact program with `go build -race` and `go run -race` reported zero data races, which confirms the mutex protects the shared slice under Colly's real concurrent callback execution. A channel that a single consumer goroutine reads from works equally well and avoids the lock, at the cost of an extra goroutine to manage. [Concurrency vs ParallelismLearn the key differences between Concurrency and Parallelism and how to leverage them in Python and JavaScript to optimize performance in various computational tasks.](https://scrapfly.io/blog/posts/concurrency-vs-parallelism) Parallel execution cuts crawl time versus a synchronous loop, since requests overlap instead of queuing. The exact speedup depends on target latency, `Parallelism`, and the machine running it. Treat any number as tied to that setup, not a fixed multiplier. Scrapfly #### Extract structured data automatically? Scrapfly's Extraction API uses AI to turn any webpage into structured data — no selectors needed. [Try Free →](https://scrapfly.io/register)## How Do You Scrape JavaScript Websites in Go? Go has real options for JavaScript-rendered pages. None of the libraries covered so far execute JavaScript, so a page that builds its content client-side needs a different approach. ### When Do You Need a Browser? Reach for a browser only when the data is missing from raw HTML, or the page needs a click or scroll to reveal it. Check first with `curl` or by viewing the page source. If the value you need is already there, you don't need a browser. If it isn't, look for a JSON or XHR endpoint the page calls before paying for a browser's overhead. Our guide to [finding hidden web data](https://scrapfly.io/blog/posts/how-to-scrape-hidden-web-data)covers that check in more depth. ### Rod, chromedp, and Playwright-Go Compared Rod is a high-level driver built directly on the Chrome DevTools Protocol, with a fluent API for navigation and element interaction. Rod's stealth features live in a separate package, [go-rod/stealth](https://github.com/go-rod/stealth). That package hasn't seen a commit since May 2023, so don't treat it as a guaranteed bypass. chromedp drives the same protocol at a lower level, through explicit actions rather than a fluent API. That makes it more verbose but more predictable for heavy automation, where you want full control over each step. Playwright-Go is a community-maintained binding, not an official Microsoft port. Microsoft's own supported language list covers TypeScript, Python, .NET, and Java, and Go isn't on it. Pick it specifically when a target needs Firefox or WebKit rendering beyond Chromium. For a broader look at browser tools across languages, see our guide to [browser automation tools](https://scrapfly.io/blog/posts/best-browser-automation-tools). ### Example: Rendering a JavaScript-Heavy Page in Go `web-scraping.dev/testimonials` ships its first batch of testimonials in the raw server HTML, so a plain `net/http` request already gets those. The rest load only when the last testimonial scrolls into view, firing an `htmx` request over XHR that no static fetch ever triggers. That's the part needing a browser: golang```golang package main import ( "context" "fmt" "log" "time" "github.com/chromedp/chromedp" ) func main() { ctx, cancel := chromedp.NewContext(context.Background()) defer cancel() ctx, cancel = context.WithTimeout(ctx, 30*time.Second) defer cancel() var loaded int var content string err := chromedp.Run(ctx, chromedp.Navigate("https://web-scraping.dev/testimonials"), chromedp.WaitVisible(`.testimonial`, chromedp.ByQuery), chromedp.ScrollIntoView(`.testimonials > *:last-child`, chromedp.ByQuery), chromedp.WaitReady(`.testimonial:nth-of-type(11)`, chromedp.ByQuery), chromedp.Evaluate(`document.querySelectorAll('.testimonial').length`, &loaded), chromedp.OuterHTML(`.testimonial:nth-of-type(11)`, &content, chromedp.ByQuery), ) if err != nil { log.Fatal(err) } fmt.Println("testimonials loaded:", loaded) fmt.Println(content) } ``` Live run: 10 testimonials on page load, 20 after the scroll. The 11th carries an `htmx-added` class, confirming it came from the scroll-triggered fetch, not the initial page. `chromedp.WaitVisible` blocks until the first batch is visible. `chromedp.WaitReady` after the scroll blocks until the 11th testimonial exists in the DOM, so neither read races a page that's still loading. See [chromedp's package documentation](https://pkg.go.dev/github.com/chromedp/chromedp)for the full set of actions available beyond navigation and waiting. ## Production Go Scraping Checklist A scraper that works once against a live target needs more guardrails before it runs unattended in production: - Bound concurrency with a worker pool or `LimitRule`, don't let goroutines run unchecked - Retry failed requests with exponential backoff, and cap the retry count - Set per-domain rate limits so a crawl doesn't look like a burst attack - Use `context.WithTimeout` on every request and respect cancellation - Cap response body size to avoid an unbounded read on a malformed response - Close response bodies promptly, and reuse the HTTP client for connection reuse - Review the target's terms of service and `robots.txt` before crawling - Log request outcomes with structured fields, beyond a pass or fail count - Write a selector snapshot test so markup drift fails the build, not production - Set a proxy rotation policy before scale forces one - Monitor extraction yield, not HTTP success rate alone. A 200 that parses to zero products is a silent failure a status-code dashboard won't catch - Version and date every dependency you pin, and revisit that pin periodically Extraction yield matters more than it sounds. A crawl logging a clean run of 200 responses while a layout change quietly zeroed out every selector looks identical to a healthy one. It stays that way until someone checks the actual output. ## Scaling Go Scrapers With Scrapfly Everything above runs against `web-scraping.dev`, a fixture with no anti-bot protection. A protected or high-scale target adds rendering, proxy management, and anti-bot bypass on top of this logic. That's where a managed layer like Scrapfly's Web Scraping API takes over. ScrapFly's [Web Scraping API](https://scrapfly.io/products/web-scraping-api) is a single endpoint for fetching public web pages with managed rendering, retries, and proxy selection. The current product page reports a **98% success rate on the hardest pages** and coverage across **190+ countries**. - [Anti-Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - configures TLS, browser profiles, proxies, and retries for protected targets through `asp=true`. - [Smart proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) - residential and datacenter pools with country and ASN level geo-targeting. - [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - render SPAs and changing pages through real cloud browsers. - [Browser automation scenarios](https://scrapfly.io/docs/scrape-api/javascript-scenario) - scroll, click, fill forms, and wait for elements without managing a browser fleet. - [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - return pages as HTML, JSON, clean text, or LLM ready Markdown. - [Session management](https://scrapfly.io/docs/scrape-api/session) - keep cookies, headers, and IPs consistent across multi step flows. - [Smart caching](https://scrapfly.io/docs/scrape-api/getting-started#api_param_cache) - cache successful responses to cut cost on repeat scraping jobs. - [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Scrapy](https://scrapfly.io/docs/sdk/scrapy), and [no-code integrations](https://scrapfly.io/docs/integration/getting-started) including [Make](https://scrapfly.io/integration/make), [n8n](https://scrapfly.io/integration/n8n), [Zapier](https://scrapfly.io/integration/zapier), [LangChain](https://scrapfly.io/integration/langchain), and [LlamaIndex](https://scrapfly.io/integration/llamaindex). The [Go SDK](https://scrapfly.io/docs/sdk/golang) wraps that same endpoint in native Go types. The request, error handling, and response parsing all stay idiomatic Go instead of a hand-built query string. The key comes from an environment variable, never a literal in the source: golang```golang package main import ( "fmt" "log" "os" "github.com/scrapfly/go-scrapfly" ) func main() { key := os.Getenv("SCRAPFLY_API_KEY") if key == "" { log.Fatal("SCRAPFLY_API_KEY is not set") } client, err := scrapfly.New(key) if err != nil { log.Fatal(err) } result, err := client.Scrape(&scrapfly.ScrapeConfig{ URL: "https://web-scraping.dev/product/1", ASP: true, RenderJS: true, Country: "us", }) if err != nil { log.Fatal(err) } fmt.Println(result.Result.Content) } ``` `ScrapeConfig` mirrors the parameters this guide has used directly: `ASP` turns on anti-bot handling, `RenderJS` swaps in a real browser, and `Country` sets the proxy's geography. `result.Result.Content` returns the rendered page's HTML, the same type this guide has been handing to goquery all along. The parsing code already built here doesn't have to change. See the [Go onboarding guide](https://scrapfly.io/docs/onboarding/golang) for a full walkthrough beyond this minimal example. ### Power your scraping with Scrapfly Forget about getting blocked. Scrapfly handles anti-bot bypasses, browser rendering, and proxy rotation so you can focus on the data. [Try for FREE!](https://scrapfly.io/register) ## FAQ What is the best Go library for web scraping?The best choice depends on the target, not a single library. Use net/http with goquery for small static jobs, Colly for crawling at scale, and Rod or chromedp when the page needs JavaScript. Pick Ferret when extraction rules should live outside your Go code. What's the difference between Colly and goquery?goquery is an HTML parser with jQuery-style selectors, and it never fetches anything on its own. Colly is a crawling framework that handles fetching, queueing, concurrency, and rate limits, and uses goquery internally for selection. How do you scrape JavaScript-rendered websites in Go?Run a real browser with Rod or chromedp for Chromium, or the community-maintained Playwright-Go for cross-browser coverage. Check whether the data loads from a JSON endpoint first, since that's usually cheaper than a browser. Is Go better than Python for web scraping?Go compiles to a single binary and handles concurrency with less memory per worker than Python's thread-based models. Python has a much larger scraping ecosystem and faster prototyping, so the better choice depends on the project. How do you avoid getting blocked with a Go scraper?Realistic headers, sane rate limits, and proxy rotation get a Go scraper past basic bot checks. Our [guide to bypassing anti-bot protection](https://scrapfly.io/blog/posts/how-to-bypass-anti-bot-protection-when-web-scraping) covers that in more depth. TLS fingerprinting and JavaScript challenges need more than header changes, which our [TLS fingerprinting guide](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-tls) covers. Is web scraping legal in Go?Scraping publicly available data is generally legal. You still need to respect a site's terms of service, robots.txt, and privacy laws covering what you collect. That analysis depends on the target and the data, not the language doing the scraping. ## Summary Start with net/http and goquery for a single static page, where you keep full control over retries and rate limits. Move to Colly once a job means crawling many pages, since it adds concurrency, per-domain limits, and caching. A hand-rolled loop would otherwise need to build all of that. Reach for Rod or chromedp when the data isn't in the raw HTML, and Playwright-Go when a target needs Firefox or WebKit. None of the libraries in this guide, goquery, Colly, or chromedp, handle proxy management or anti-bot bypass. That stays your problem once a target gets protected or scales up. A managed layer like Scrapfly's Web Scraping API then takes over fetching, while your Go parsing code keeps working unchanged. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [Which Go Web Scraping Library Should You Use?](#which-go-web-scraping-library-should-you-use) - [Static HTML: net/http, goquery, and htmlquery](#static-html-net-http-goquery-and-htmlquery) - [Crawling Frameworks: Colly](#crawling-frameworks-colly) - [JavaScript Rendering: Rod, chromedp, and Playwright-Go](#javascript-rendering-rod-chromedp-and-playwright-go) - [Declarative Extraction: Ferret](#declarative-extraction-ferret) - [Setup: Go Environment and Packages](#setup-go-environment-and-packages) - [Setting Up the Go Environment](#setting-up-the-go-environment) - [Installing the Packages](#installing-the-packages) - [Web Scraping With Go: Core Concepts](#web-scraping-with-go-core-concepts) - [Sending HTTP Requests](#sending-http-requests) - [Parsing HTML With goquery (CSS Selectors)](#parsing-html-with-goquery-css-selectors) - [Parsing HTML With htmlquery (XPath)](#parsing-html-with-htmlquery-xpath) - [Crawling Links Between Pages](#crawling-links-between-pages) - [Example Go Scraper: net/http and goquery](#example-go-scraper-net-http-and-goquery) - [Web Scraping With Colly](#web-scraping-with-colly) - [How Colly Works: Collectors](#how-colly-works-collectors) - [How Colly Works: Callbacks](#how-colly-works-callbacks) - [Example Colly Scraper](#example-colly-scraper) - [Rate Limits, Retries, and Error Handling](#rate-limits-retries-and-error-handling) - [Using Proxies With Colly](#using-proxies-with-colly) - [Concurrency and Safe Result Collection](#concurrency-and-safe-result-collection) - [How Do You Scrape JavaScript Websites in Go?](#how-do-you-scrape-javascript-websites-in-go) - [When Do You Need a Browser?](#when-do-you-need-a-browser) - [Rod, chromedp, and Playwright-Go Compared](#rod-chromedp-and-playwright-go-compared) - [Example: Rendering a JavaScript-Heavy Page in Go](#example-rendering-a-javascript-heavy-page-in-go) - [Production Go Scraping Checklist](#production-go-scraping-checklist) - [Scaling Go Scrapers With Scrapfly](#scaling-go-scrapers-with-scrapfly) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ ### Crawl4AI Guide: Web Crawling for LLMs, RAG, and AI Agents Learn how to use Crawl4AI v0.8.x for AI-ready web crawling.Covers installation, LLM extraction with Pydantic, deep crawl... ](https://scrapfly.io/blog/posts/crawl4AI-explained) [ python ### How to Crawl the Web with Python Introduction to web crawling with Python. What is web crawling? How it differs from web scraping? And a deep dive into c... ](https://scrapfly.io/blog/posts/crawling-with-python) [ python ai ### What is Parsing? From Raw Data to Insights Learn about the fundamentals of parsing data, across formats like JSON, XML, HTML, and PDFs. Learn how to use Python par... ](https://scrapfly.io/blog/posts/what-is-parsing-turning-data-into-insights) ## Related Questions - [ Q What's the difference between Web Scraping and Crawling? ](https://scrapfly.io/blog/answers/whats-the-difference-between-scraping-and-crawling) - [ Q How to ignore non HTML URLs when web crawling? ](https://scrapfly.io/blog/answers/how-to-ignore-non-html-urls-when-web-crawling) Extract structured data with AI, **1,000 free credits** [Start Free](https://scrapfly.io/register)