[Blog](https://scrapfly.io/blog)   /  [api](https://scrapfly.io/blog/tag/api)   /  [How to Scrape DigiKey Electronic Component Data in 2026](https://scrapfly.io/blog/posts/how-to-scrape-digikey)   # How to Scrape DigiKey Electronic Component Data in 2026

 by [Mayada Shaaban](https://scrapfly.io/blog/author/mayada-shaaban-90143e67) Aug 13, 2026 22 min read [\#api](https://scrapfly.io/blog/tag/api) [\#blocking](https://scrapfly.io/blog/tag/blocking) [\#python](https://scrapfly.io/blog/tag/python) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) 

 [  ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-digikey "Share on LinkedIn") [  ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-digikey&text=How%20to%20Scrape%20DigiKey%20Electronic%20Component%20Data%20in%202026 "Share on X") [  ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-digikey "Share on Facebook")    

 

 

Summarize this article with

 [  ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-digikey) [  ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-digikey) [  ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-digikey) [  ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-digikey) [  ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-digikey) 



         

Anyone who has built a bill of materials tool knows the feeling. You think the parts list is final, then a big buyer empties stock on a specialty IC overnight. Now you jump between DigiKey, Mouser, and Newark hoping something is still available.

That pain is exactly what engineers describe in the [r/embedded thread](https://www.reddit.com/r/embedded/comments/1qk1gtp/normalizing_pricing_stock_across_mouser_digikey/) on normalizing pricing and stock across distributors (January 2026).

DigiKey lists over 18 million components with live price breaks, stock, lead times, and parametric specs. A plain `requests.get()` against a product page then returns a Cloudflare challenge and a 403, not component HTML.

This guide shows how to scrape DigiKey reliably in 2026. You will see which surfaces yield clean data and how to get past Cloudflare without tricks. It also covers when the official API beats scraping, and how Mouser fits a multi-distributor setup.

[**Latest DigiKey Scraper Code**github.com/scrapfly/scrapfly-scrapers/digikey-scraper](https://github.com/scrapfly/scrapfly-scrapers/tree/main/digikey-scraper)



## Key Takeaways

- **Three public surfaces carry the data:** category listings, keyword search, detail pages.
- **Every scraped surface exposes embedded JSON:** the parser reads `script#__NEXT_DATA__` instead of maintaining selectors for visible product rows.
- **Detail records carry structure:** price-break tiers, stock status, and lifecycle flags.
- **Cloudflare is the one hard part:** a plain request gets a 403; the JSON parses easily.
- **Baseline tricks do not work:** Cloudflare checks TLS fingerprints before the User-Agent.
- **The API v4 covers search and detail lookups:** KeywordSearch supports keywords, manufacturer/category restrictions, and parametric filters; ProductDetails handles a single product number.

**Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime.







## Why Scrape DigiKey?

DigiKey's public catalog is useful when a workflow needs repeated snapshots or one extraction schema across distributors. Common jobs include:

- Competitive price-break monitoring across many components at once
- BOM cost estimation and multi-distributor sourcing
- Real-time stock and lead-time tracking, to catch stock dumps before checkout
- Obsolescence monitoring through lifecycle status (NRND or Last Time Buy)
- Parametric catalog data collection for part selection

These are not hypothetical. In the r/embedded thread, engineers describe jumping between DigiKey, Mouser, and Newark when a large purchaser empties stock on a specialty IC. They want scripts that check stock and price ahead of time, not at checkout.

The decision below shows when scraping is the right path and when DigiKey's API v4 fits instead. Most jobs land on scraping:

| Use case | Path |
|---|---|
| Bulk catalog export or database creation | Review DigiKey's terms; its API agreement prohibits bulk downloads and using API data to build a database |
| Keyword search across manufacturers | DigiKey API v4 or scrape |
| Parametric filtering for part selection | DigiKey API v4 or scrape |
| Multi-distributor BOM tools (DigiKey, Mouser, TME, Newark) | Either; the API adds DigiKey-specific auth and schema handling |
| Volume above the Standard API quota | Scrape only if the use complies with DigiKey's site terms |
| Public-page fields absent from the API response | Scrape |
| Exact product pricing and availability | DigiKey ProductDetails |
| Account-specific pricing | DigiKey ProductDetails or ProductPricing |

The API and scraping overlap on search and product data. Choose between them based on quota, data freshness, use terms, and whether you need one extraction path across distributors.



## When the DigiKey API Fits (and Where It Doesn't)

DigiKey publishes a [Product Information API v4](https://developer.digikey.com/products/product-information-v4) with KeywordSearch, ProductDetails, and ProductPricing endpoints.

The API covers both discovery and exact-product workflows. KeywordSearch accepts keywords plus manufacturer, category, and parametric filters, with pagination of up to 50 products per request. ProductDetails and ProductPricing resolve a product number and can return account-specific pricing.

Use the API when these conditions hold:

- You can complete OAuth2 onboarding and keep the client credentials secure
- The Standard Product Information quota of 120 requests per minute and 1,000 per day fits the workload
- KeywordSearch data that may be up to 24 hours stale is acceptable, or you can use ProductDetails for real-time pricing and availability
- Your use complies with DigiKey's API agreement

Scraping remains useful when you need the public page payload, fields that are absent from the API response, or one parser contract across several distributors. It is not a way around DigiKey's rules: the site terms prohibit automated content gathering, while the API agreement prohibits bulk downloads, database creation from API data, and competitive use.

DigiKey's developer portal owns the OAuth2 implementation details. This guide covers the public-page extraction path without pretending the API lacks search or parametric filtering.



## Why Is DigiKey Hard to Scrape?

DigiKey's product and search pages are not hard to parse, since the data is all there in the HTML. The hard part is getting the page at all, because DigiKey runs a Cloudflare managed challenge on product and search pages.

A plain `requests.get()` returns HTTP 403 with a Cloudflare interstitial, not component HTML. We reproduced this on August 13, 2026: the response included `server: cloudflare`, `cf-mitigated: challenge`, `cf-ray`, and a `Just a moment` body.

Cloudflare checks several signals before it serves content. It inspects the TLS and JA3 fingerprint, the HTTP/2 fingerprint, and the browser fingerprint with client hints (the `Sec-CH-UA-*` headers). It also weighs IP reputation, so the same browser fingerprint can get a different result from another network.

Cloudflare serves a JavaScript challenge. The blocked response already sets `__cf_bm`, so that cookie is not proof of a cleared session; a passed challenge may set `cf_clearance`.

These are the failure modes you will hit, and why each one fails:

- **Plain HTTP returns a 403 challenge.** `requests` and `httpx` get `cf-mitigated: challenge` and a Cloudflare interstitial, not HTML.
- **User-Agent rotation does not help.** Cloudflare fingerprints the TLS handshake before it reads the UA string, so `requests` is distinguishable from Chrome on request one.
- **Brittle selectors fail independently of Cloudflare.** The cited 2021 Selenium question failed to find the stock element, and its accepted answer changed the XPath. It does not establish an anti-bot block.
- **IP reputation changes the result.** If one exit is challenged repeatedly, test another network before changing the parser.

In our August 13 test, a browser-rendered ASP request returned HTTP 200 with the DigiKey product page that plain curl could not reach.

The

[How to Bypass Cloudflare When Web Scraping in 2026Cloudflare offers one of the most popular anti scraping service, so in this article we'll take a look how it works and how to bypass it.](https://scrapfly.io/blog/posts/how-to-bypass-cloudflare-anti-scraping)

 covers the deeper technique. The [TLS fingerprinting guide](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-tls) explains why the handshake gives a plain client away. [How to Fix 403 Forbidden Errors When Web ScrapingLearn why web scrapers get 403 Forbidden errors and how to fix them with 7 Python solutions, from headers to TLS fingerprinting.](https://scrapfly.io/blog/posts/403-forbidden-web-scraping)



## Setting Up the Scraper

You need Python 3.10 or newer, the `scrapfly-sdk` client, `loguru` for readable logs, and `lzstring` to build DigiKey's compressed pagination cursor. Install all three with a single command:

bash```bash
pip install scrapfly-sdk loguru lzstring
```



Set your Scrapfly key as an environment variable so it stays out of your code:

bash```bash
export SCRAPFLY_KEY="YOUR_SCRAPFLY_KEY"
```



Every request in this guide reuses one config, and every parser reads the same embedded JSON blob:

python```python
import os
import json
from typing import Dict, List, Optional
from urllib.parse import quote
from loguru import logger as log
from lzstring import LZString
from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse, ScrapflyScrapeError

SCRAPFLY = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"])
_LZSTRING = LZString()

BASE_CONFIG = {
    "asp": True,         # enable ASP routing when the target challenges the request
    "render_js": True,   # render DigiKey's dynamic Next.js page
    "country": "us",     # request a US exit; ASP selects the bypass route
}


def _page_data(sel) -> tuple[Dict, Dict]:
    """DigiKey is a Next.js app: every surface ships its data in __NEXT_DATA__."""
    raw = sel.css("script#__NEXT_DATA__::text").get() or "{}"
    try:
        next_data = json.loads(raw)
    except json.JSONDecodeError:
        next_data = {}
    props = next_data.get("props", {})
    data = props.get("pageProps", {}).get("envelope", {}).get("data", {}) or {}
    return data, props


def _absolute_url(url: Optional[str]) -> Optional[str]:
    if not url:
        return url
    if url.startswith("//"):
        return f"https:{url}"
    if url.startswith("/"):
        return f"https://www.digikey.com{url}"
    return url
```



That `_page_data` helper is the whole trick. DigiKey runs on Next.js, so listings, search, and detail pages all serialize their state into a `<script id="__NEXT_DATA__">` tag. You get JSON instead of CSS selectors to maintain against layout changes.

The full maintained module lives in [scrapfly-scrapers/digikey-scraper](https://github.com/scrapfly/scrapfly-scrapers/tree/main/digikey-scraper) on GitHub. The Scrapfly SDKs also cover TypeScript, Go, and Rust, plus a Scrapy integration, so non-Python stacks can follow the same approach. Start with category listings, which expose part links and the compressed pagination cursor.



## How to Scrape DigiKey Parametric Search Results

DigiKey exposes catalog listings two ways: parametric category pages at `/en/products/filter/{category}/{id}` and keyword search at `/en/products/result?keywords=...`. Both return rows you can iterate to collect part links, then pull full records from the detail pages.

### Scraping category listings

Category rows arrive as a `products` array of loosely typed cells, so the parser identifies each cell by the keys it carries rather than by position:

python```python
def parse_category(response: ScrapeApiResponse) -> List[Dict]:
    sel = response.selector
    data, props = _page_data(sel)
    currency = props.get("currency") or "USD"
    rows = data.get("products") or []

    results = []
    for row in rows:
        compare, detail, price_entries, qty_entries = {}, {}, [], []
        for cell in row:
            if not isinstance(cell, dict):
                continue
            value = cell.get("value")
            if isinstance(value, dict):
                if "detailUrl" in value:
                    detail = value
                elif "productNumber" in value:
                    compare = value
            elif isinstance(value, list) and value and isinstance(value[0], dict):
                if "unitPrice" in value[0]:
                    price_entries = value
                elif "quantity" in value[0]:
                    qty_entries = value

        stock_quantity, availability = None, None
        if qty_entries:
            try:
                stock_quantity = int(str(qty_entries[0].get("quantity")).replace(",", ""))
            except (ValueError, TypeError):
                stock_quantity = None
            availability = qty_entries[0].get("label")

        price = compare.get("price")
        if price is None and price_entries:
            raw_price = price_entries[0].get("unitPrice")
            try:
                price = str(float(raw_price.lstrip("$"))) if raw_price else None
            except (ValueError, AttributeError):
                price = raw_price

        image_obj = detail.get("image") or {}
        manufacturer_obj = compare.get("manufacturer")
        manufacturer = manufacturer_obj.get("Name") if isinstance(manufacturer_obj, dict) else manufacturer_obj

        results.append({
            "name": detail.get("description"),
            "url": _absolute_url(detail.get("detailUrl")),
            "digikey_part_number": compare.get("productNumber"),
            "manufacturer_part_number": compare.get("manufacturerPartNumber"),
            "manufacturer": manufacturer,
            "price": price,
            "currency": currency,
            "stock_quantity": stock_quantity,
            "availability": availability,
            "image": _absolute_url(image_obj.get("standard") or image_obj.get("thumb")),
        })

    return results
```



Pagination is the one non-obvious part. DigiKey encodes page state in an `s=` query parameter holding an LZ-String-compressed JSON payload. You build that cursor rather than incrementing a page number:

python```python
async def scrape_category(url: str, max_pages: int = 3) -> List[Dict]:
    log.info("scraping category {}", url)
    first_page = await SCRAPFLY.async_scrape(ScrapeConfig(url, **BASE_CONFIG))
    results = parse_category(first_page)

    if max_pages > 1:
        to_scrape = []
        for page in range(2, max_pages + 1):
            payload = json.dumps({"5": {"p": page, "pp": 25}}, separators=(",", ":"))
            cursor = quote(_LZSTRING.compressToEncodedURIComponent(payload), safe="")
            sep = "&" if "?" in url else "?"
            to_scrape.append(ScrapeConfig(f"{url}{sep}s={cursor}", **BASE_CONFIG))
        async for response in SCRAPFLY.concurrent_scrape(to_scrape):
            if isinstance(response, ScrapflyScrapeError):
                log.error("failed to scrape category page: {}", response.error)
                continue
            try:
                results.extend(parse_category(response))
            except Exception as e:
                log.error("failed to parse category page: {}", e)

    log.success("scraped {} category results", len(results))
    return results
```



The `{"5": {"p": page, "pp": 25}}` payload sets the page number and page size. Scrapfly then fetches every page after the first concurrently. Running it against the industrial automation accessories category returns rows like these:

json```json
[
  {
    "name": "UNIVERSAL WALL ADAPTER",
    "url": "https://www.digikey.com/en/products/detail/phoenix-contact/2938235/2553505",
    "digikey_part_number": "277-6985-ND",
    "manufacturer_part_number": "2938235",
    "manufacturer": "Phoenix Contact",
    "price": "40.73",
    "currency": "USD",
    "stock_quantity": 119,
    "availability": "In Stock",
    "image": "https://mm.digikey.com/Volume0/opasdata/d220001/medias/images/401/2938235.JPG"
  },
  {
    "name": "RS RJ 45 8 POLE INTERFACE",
    "url": "https://www.digikey.com/en/products/detail/weidmüller/8611320000/4027219",
    "digikey_part_number": "281-3253-ND",
    "manufacturer_part_number": "8611320000",
    "manufacturer": "Weidmüller",
    "price": "41.58",
    "currency": "USD",
    "stock_quantity": 614,
    "availability": "In Stock",
    "image": "https://mm.digikey.com/Volume0/opasdata/d220001/medias/images/729/MFG_8611320000.JPG"
  }
]
```



### Scraping keyword search

Keyword search returns a different shape. Instead of one flat table it splits into `exactMatch` parts, `topResults` categories, and a nested `categories` tree, so the parser flattens all three into one result list:

python```python
def parse_search(response: ScrapeApiResponse) -> List[Dict]:
    data, _ = _page_data(response.selector)
    results = []

    for item in data.get("exactMatch") or []:
        raw_price = item.get("unitPrice")
        try:
            price = str(float(raw_price.lstrip("$"))) if raw_price else None
        except (ValueError, AttributeError):
            price = raw_price
        entry = {
            "name": item.get("description"),
            "url": _absolute_url(item.get("detailUrl")),
            "manufacturer_part_number": item.get("mfrProduct"),
            "manufacturer": item.get("mfr"),
            "price": price,
            "currency": "USD" if price else None,
            "image": _absolute_url(item.get("imageUrl")),
        }
        results.append({k: v for k, v in entry.items() if v is not None})

    for item in data.get("topResults") or []:
        try:
            stock = int(str(item["productCount"]).replace(",", ""))
        except (ValueError, KeyError, TypeError):
            stock = None
        entry = {
            "name": item.get("categoryName"),
            "url": _absolute_url(item.get("categoryUrl")),
            "manufacturer": item.get("parentCategory"),
            "stock_quantity": stock,
            "image": _absolute_url(item.get("imageUrl")),
        }
        results.append({k: v for k, v in entry.items() if v is not None})

    # walk the nested category tree breadth-first
    pending = [(data.get("categories") or [], None)]
    while pending:
        nodes, parent = pending.pop(0)
        for node in nodes:
            if node.get("productCount") is not None:
                try:
                    stock = int(str(node["productCount"]).replace(",", ""))
                except (ValueError, TypeError):
                    stock = None
                entry = {
                    "name": node.get("label"),
                    "url": _absolute_url(node.get("url")),
                    "manufacturer": parent,
                    "stock_quantity": stock,
                }
                results.append({k: v for k, v in entry.items() if v is not None})
            if node.get("subCategories"):
                pending.append((node["subCategories"], node.get("label") or parent))

    return results


async def scrape_search(keywords: str) -> List[Dict]:
    url = f"https://www.digikey.com/en/products/result?keywords={quote(keywords)}"
    log.info("scraping search results for keywords: {}", keywords)
    response = await SCRAPFLY.async_scrape(ScrapeConfig(url, **BASE_CONFIG))
    results = parse_search(response)
    log.success(f"scraped {len(results)} search results")
    return results
```



A broad term like "Power Transformers" resolves mostly to category nodes with product counts, which is the natural entry point for a full catalog walk:

json```json
[
  {
    "name": "Power Transformers",
    "url": "https://www.digikey.com/en/products/filter/power-transformers/164",
    "manufacturer": "Transformers",
    "stock_quantity": 6903
  },
  {
    "name": "Switching Converter, SMPS Transformers",
    "url": "https://www.digikey.com/en/products/filter/switching-converter-smps-transformers/168",
    "manufacturer": "Transformers",
    "stock_quantity": 289
  }
]
```



Both surfaces give you detail URLs. Those detail pages hold the full record.



Scrapfly

#### Need to bypass anti-bot protection?

Scrapfly's Anti-Scraping Protection handles Cloudflare, DataDome, and more — automatically.

[Try Free →](https://scrapfly.io/register)## How to Scrape DigiKey Product Detail Pages

DigiKey product detail pages live at `/en/products/detail/{manufacturer}/{part-number}/{id}` and carry the complete record for a part.

Detail pages are the surface BOM and price-monitoring tools care about most: pricing, stock, the full spec table, and lifecycle status in one place.

Detail pages carry two data sources worth reading. The `__NEXT_DATA__` envelope holds the structured blocks (`productOverview`, `quantityTable`, `productAttributes`, `environmental`), and a JSON-LD `@graph` block provides a fallback for the core commerce fields:

python```python
def parse_product(response: ScrapeApiResponse) -> Dict:
    sel = response.selector
    data, _ = _page_data(sel)

    product_ld = {}
    for script in sel.css('script[type="application/ld+json"]'):
        try:
            graph = json.loads(script.css("::text").get() or "{}").get("@graph", [])
        except json.JSONDecodeError:
            continue
        product_ld = next((node for node in graph if node.get("@type") == "Product"), {})
        if product_ld:
            break

    offers = product_ld.get("offers", {})
    overview = data.get("productOverview", {}) or {}
    price_quantity = data.get("priceQuantity", {}) or {}
    quantity_table = data.get("quantityTable") or []
    attrs_block = data.get("productAttributes", {}) or {}
    attributes = attrs_block.get("attributes") or []
    categories = attrs_block.get("categories") or []
    environmental = data.get("environmental", {}) or {}
    carousel_media = data.get("carouselMedia") or []
    messages = data.get("messages") or []
```



Keeping both sources means a missing block on one part does not empty the record. The rest of the function pulls the two groups of fields that matter.

### Extracting price breaks and stock

The pricing and availability block holds the numbers behind cost estimation: price break tiers, current stock, lead time, and minimum order quantity.

python```python
    price_breaks = [
        {"quantity": tier["breakQty"], "unit_price": tier["unitPrice"]}
        for tier in quantity_table
        if tier.get("breakQty") is not None and tier.get("unitPrice") is not None
    ]

    if quantity_table:
        price = str(quantity_table[0]["unitPrice"])
    elif offers.get("price") is not None:
        price = str(offers["price"])
    else:
        price = None

    availability = next(
        (m.get("message", "").strip() for m in messages if m.get("type") == "title"),
        None,
    )
    if not availability and offers.get("availability"):
        availability = offers["availability"].replace("https://schema.org/", "")

    stock_quantity = None
    qty_available = price_quantity.get("qtyAvailable")
    if qty_available is not None:
        try:
            stock_quantity = int(str(qty_available).replace(",", ""))
        except ValueError:
            pass

    pricing_list = price_quantity.get("pricing") or []
    min_order_qty = pricing_list[0].get("minOrderQuantity") if pricing_list else None
```



Note that `quantityTable` lists the best price first, so `quantity_table[0]` is the highest-volume tier, not the single-unit price. These fields are the snapshot a monitoring run compares between fetches.

### Extracting parametric specs and lifecycle status

The rest of the page describes what the part is and whether you should design it in. Specs come from `productAttributes` as labelled values, and lifecycle status is one of those attributes, labelled `Part Status`:

python```python
    specifications = {}
    lifecycle_status = None
    for attr in attributes:
        label = attr.get("label")
        values = attr.get("values") or []
        value = ", ".join(v.get("value", "") for v in values if v.get("value"))
        if label and value:
            specifications[label] = value
            if label == "Part Status":
                lifecycle_status = value

    env_map = {}
    for row in environmental.get("dataRows") or []:
        cells = row.get("dataCells") or []
        if len(cells) >= 2:
            label = cells[0].get("data", {}).get("value", {}).get("value")
            value = cells[1].get("data", {}).get("value", {}).get("value")
            if label and value:
                env_map[label] = value

    image = next(
        (
            _absolute_url(media["displayUrl"])
            for media in carousel_media
            if media.get("type") == "Image" and media.get("displayUrl")
        ),
        None,
    )

    return {
        "name": overview.get("title") or product_ld.get("name") or "",
        "url": response.context.get("url", ""),
        "digikey_part_number": overview.get("rolledUpProductNumber") or product_ld.get("sku") or "",
        "manufacturer_part_number": overview.get("manufacturerProductNumber") or product_ld.get("mpn"),
        "manufacturer": overview.get("manufacturer") or (product_ld.get("brand") or {}).get("name"),
        "description": overview.get("detailedDescription") or product_ld.get("description"),
        "category": categories[-1].get("label") if categories else None,
        "price": price,
        "currency": offers.get("priceCurrency") or "USD",
        "price_breaks": price_breaks or None,
        "stock_quantity": stock_quantity,
        "availability": availability,
        "lead_time": overview.get("standardLeadTime"),
        "min_order_qty": min_order_qty,
        "lifecycle_status": lifecycle_status,
        "rohs_status": env_map.get("RoHS Status"),
        "reach_status": env_map.get("REACH Status"),
        "msl_rating": env_map.get("Moisture Sensitivity Level (MSL)"),
        "datasheet_url": _absolute_url(overview.get("datasheetUrl")),
        "image": image,
        "specifications": specifications,
    }


async def scrape_products(urls: List[str]) -> List[Dict]:
    to_scrape = [ScrapeConfig(url, **BASE_CONFIG) for url in urls]
    products = []
    async for response in SCRAPFLY.concurrent_scrape(to_scrape):
        if isinstance(response, ScrapflyScrapeError):
            log.error("failed to scrape product: {}", response.error)
            continue
        try:
            products.append(parse_product(response))
        except Exception as e:
            log.error("failed to parse product: {}", e)
    log.success("scraped {} products", len(products))
    return products
```



A single part comes back as one complete record, trimmed here for length:

json```json
{
  "name": "F16-150-C2",
  "url": "https://www.digikey.com/en/products/detail/triad-magnetics/F16-150-C2/3986399",
  "digikey_part_number": "237-F16-150-C2-ND",
  "manufacturer_part_number": "F16-150-C2",
  "manufacturer": "Triad Magnetics",
  "category": "Power Transformers",
  "price": "4.27391",
  "currency": "USD",
  "price_breaks": [
    { "quantity": 1000, "unit_price": 4.27391 },
    { "quantity": 100, "unit_price": 4.7501 },
    { "quantity": 10, "unit_price": 5.27 },
    { "quantity": 1, "unit_price": 5.83 }
  ],
  "stock_quantity": 2111,
  "availability": "In-Stock: 2,111",
  "lead_time": "19 Weeks",
  "min_order_qty": 1,
  "lifecycle_status": "Active",
  "rohs_status": "ROHS3 Compliant",
  "reach_status": "REACH Unaffected",
  "msl_rating": "Not Applicable",
  "datasheet_url": "http://catalog.triadmagnetics.com/Asset/F16-150-C2.pdf",
  "specifications": {
    "Part Status": "Active",
    "Type": "Laminated Core",
    "Voltage - Primary": "115V",
    "Voltage - Secondary (Full Load)": "Parallel 8V, Series 16V",
    "Power - Max": "2.5VA",
    "Mounting Type": "Through Hole"
  }
}
```



Lifecycle status powers obsolescence monitoring, a real procurement job: catching an NRND flag during a BOM run is far cheaper than discovering it at reorder. Once you can pull this record, the next problem is reconciling it with other distributors.



## How Do You Normalize DigiKey Data Across a Multi-Distributor BOM?

If you are pulling DigiKey alongside Mouser, TME, and Newark, the hard part is not any single scraper, it is reconciling them into one schema.

Engineers describe this directly in the r/embedded thread: "different auth and rate limits, inconsistent price tiers, stock vs lead-time semantics." The fix is a single normalized record that every distributor scraper maps into.

A workable record covers the fields that matter for comparison and caching:

python```python
from dataclasses import dataclass, field
from datetime import datetime, timezone

@dataclass
class DistributorPart:
    mpn: str
    distributor: str                       # digikey / mouser / tme / newark
    price_breaks: list = field(default_factory=list)  # [{"qty": 1, "price": 0.10}]
    stock_qty: int = 0
    stock_status: str = "Unknown"          # In Stock / Back Order / Out of Stock
    lead_time_weeks: int | None = None
    lifecycle_status: str = "Active"       # Active / NRND / Obsolete / Last Time Buy
    currency: str = "USD"
    source_url: str = ""
    fetched_at: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())
```



One schema gives you a single comparison layer across every source. Engineers also cache aggressively, often for 15 to 60 minutes, to stay within limits, which keeps BOM pricing usable in scripts.

The

[How to Use Cache In Web Scraping for Major Performance BoostIntroduction to web scraping caches. How caching can significantly reduce scraping costs and drastically improve performance.](https://scrapfly.io/blog/posts/how-to-use-cache-in-web-scraping)

 covers the trade-offs. Scrapfly's caching cuts repeat-fetch cost when you snapshot the same parts. Normalization starts after the fetch succeeds; DigiKey's Cloudflare challenge is the remaining fetch-layer constraint.

## Bypassing Cloudflare with Scrapfly

A plain request to DigiKey returned HTTP 403 with `server: cloudflare`, `cf-mitigated: challenge`, `cf-ray`, and a `Just a moment` body in our August 13 check. Changing the User-Agent does not make that response parseable.

The maintained scraper uses the same three-key configuration for product, category, and search pages:

- `asp=True` lets Scrapfly choose the supported route when protection is encountered.
- `render_js=True` loads DigiKey's dynamic Next.js page.
- `country="us"` requests the US catalog.

In fresh product, category, and search runs, that configuration returned HTTP 200 and the page payload needed by `_page_data()`. This result proves the configured request worked. It does not establish that residential proxies are required or identify which internal route handled each request.

IP reputation can still change an individual result. If one exit is challenged repeatedly, test another network before changing the parser, but do not treat one proxy class as a universal requirement without a controlled comparison.

For the protection model and current product route, read Scrapfly's [Cloudflare bypass page](https://scrapfly.io/bypass/cloudflare).



## Scrape DigiKey Data with Scrapfly



ScrapFly's [Web Scraping API](https://scrapfly.io/products/web-scraping-api) combines anti-bot handling, JavaScript rendering, proxy routing, sessions, caching, and SDKs behind one endpoint.

- [Anti-Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - lets the service select a supported route when a target challenges the request.
- [Proxy routing](https://scrapfly.io/docs/scrape-api/proxy) - provides residential and datacenter pools with country and ASN geo-targeting.
- [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - renders SPAs and JavaScript-heavy pages through cloud browsers.
- [Browser automation scenarios](https://scrapfly.io/docs/scrape-api/javascript-scenario) - scrolls, clicks, fills forms, and waits for elements without a browser fleet.
- [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - returns HTML, JSON, clean text, or Markdown.
- [Session management](https://scrapfly.io/docs/scrape-api/session) - keeps cookies, headers, and IPs consistent across multi-step flows.
- [Caching](https://scrapfly.io/docs/scrape-api/getting-started#api_param_cache) - caches successful responses for repeated jobs.
- [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Scrapy](https://scrapfly.io/docs/sdk/scrapy), and [integrations](https://scrapfly.io/docs/integration/getting-started) cover common application stacks.



### Power your scraping with Scrapfly

Forget about getting blocked. Scrapfly handles anti-bot bypasses, browser rendering, and proxy rotation so you can focus on the data.



[Try for FREE!](https://scrapfly.io/register)



## FAQ

What does the DigiKey API require?You need a My DigiKey account, a developer-portal app, and OAuth2 credentials. Standard Product Information access is limited to 120 requests per minute and 1,000 per day; confirm any commercial terms in your account before building around it.







Is it legal to scrape DigiKey?There is no universal yes-or-no answer. In *hiQ Labs v. LinkedIn* (9th Circuit, 2022), the court held that accessing publicly available profiles was not "without authorization" under the CFAA, but DigiKey's Terms of Use prohibit robots, scrapers, spiders, automated content gathering, and bypassing access restrictions. Review those terms and the law that applies to your use case; this is not legal advice.







Why does my DigiKey scraper get 403 errors?Because a plain `requests` call currently receives DigiKey's Cloudflare challenge before any component HTML. In our test, the browser-rendered Scrapfly request returned the page. Do not assume residential routing is mandatory without a controlled proxy-pool comparison.







Can I track DigiKey price and stock changes over time?DigiKey returns current snapshot pricing and stock only, and neither the site nor the API exposes history. To track changes, run regular snapshots yourself and store them, then compare price-break tiers and stock between runs.







Can I detect obsolete or NRND parts when scraping DigiKey?Yes, DigiKey product detail pages carry a lifecycle status field (Active, NRND for "Not Recommended for New Designs", Obsolete, or Last Time Buy). Capturing it lets you flag end-of-life parts during BOM monitoring instead of finding out at reorder.









## Summary

DigiKey publishes structured data across parametric listings, keyword search, and detail pages, all behind a Cloudflare challenge. Every surface is a Next.js page, so the data comes out of the `__NEXT_DATA__` JSON blob instead of brittle CSS selectors. Detail pages carry the richest record: price breaks, stock, the spec table, and lifecycle status. The blocker is Cloudflare, not the markup.

Baseline User-Agent tricks do not bypass that challenge. API v4 supports keyword, category, and parametric search as well as single-product detail; its constraints are OAuth2, quota, data freshness, and use terms.

With `asp=True`, `render_js=True`, and `country="us"`, Scrapfly returns the page before `_page_data()` reads `__NEXT_DATA__`.

For repeated DigiKey fetches, ASP handles the browser and proxy layer while the same parser consumes `__NEXT_DATA__`. Pair this guide with the Mouser companion and DigiKey's developer portal for the API integration path.



Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect:

- Do not scrape at rates that could damage the website.
- Do not scrape data that's not available publicly.
- Do not store PII of EU citizens protected by GDPR.
- Do not repurpose *entire* public datasets which can be illegal in some countries.

Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer.

 

   [  Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents















 

  Table of Contents- [Key Takeaways](#key-takeaways)
- [Why Scrape DigiKey?](#why-scrape-digikey)
- [When the DigiKey API Fits (and Where It Doesn't)](#when-the-digikey-api-fits-and-where-it-doesn-t)
- [Why Is DigiKey Hard to Scrape?](#why-is-digikey-hard-to-scrape)
- [Setting Up the Scraper](#setting-up-the-scraper)
- [How to Scrape DigiKey Parametric Search Results](#how-to-scrape-digikey-parametric-search-results)
- [Scraping category listings](#scraping-category-listings)
- [Scraping keyword search](#scraping-keyword-search)
- [How to Scrape DigiKey Product Detail Pages](#how-to-scrape-digikey-product-detail-pages)
- [Extracting price breaks and stock](#extracting-price-breaks-and-stock)
- [Extracting parametric specs and lifecycle status](#extracting-parametric-specs-and-lifecycle-status)
- [How Do You Normalize DigiKey Data Across a Multi-Distributor BOM?](#how-do-you-normalize-digikey-data-across-a-multi-distributor-bom)
- [Bypassing Cloudflare with Scrapfly](#bypassing-cloudflare-with-scrapfly)
- [Scrape DigiKey Data with Scrapfly](#scrape-digikey-data-with-scrapfly)
- [FAQ](#faq)
- [Summary](#summary)
 
    Join the Newsletter  Get monthly web scraping insights 

 

  



Scale Your Web Scraping

Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits.

  No credit card required  1,000 free API credits  Anti-bot bypass included 

 [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) 

 Not ready? Get our newsletter instead. 

 

 ## Related Articles

 [  

 blocking 

### How to Bypass Cloudflare When Web Scraping in 2026

Cloudflare offers one of the most popular anti scraping service, so in this article we'll take a look how it works and h...

 

 ](https://scrapfly.io/blog/posts/how-to-bypass-cloudflare-anti-scraping) [     

 python scrapeguide 

### How to Scrape RS-Online (rs-online.com) in 2026

How to scrape RS-Online's North American listings and product pages for pricing, stock, specifications, and datasheet li...

 

 ](https://scrapfly.io/blog/posts/how-to-scrape-rs-online) [     

 python blocking 

### How Cloudflare Detects Bots: TLS, HTTP/2, Canvas, and Turnstile Explained

Learn how Cloudflare detects bots using TLS, HTTP/2, Canvas, WebGL, behavioral analysis, and Turnstile, plus how to scra...

 

 ](https://scrapfly.io/blog/posts/how-cloudflare-detects-bots) 

  



   



 Bypass anti-bot protection automatically, **1,000 free credits** [Start Free](https://scrapfly.io/register)