[Blog](https://scrapfly.io/blog) / [python](https://scrapfly.io/blog/tag/python) / [How to Scrape Shopify Stores for Products, Prices, and Variants](https://scrapfly.io/blog/posts/how-to-scrape-shopify-stores) # How to Scrape Shopify Stores for Products, Prices, and Variants by [Mohab Yousry](https://scrapfly.io/blog/author/mohab-yousry-9396552a) Oct 19, 2026 23 min read [\#python](https://scrapfly.io/blog/tag/python) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-shopify-stores "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-shopify-stores&text=How%20to%20Scrape%20Shopify%20Stores%20for%20Products%2C%20Prices%2C%20and%20Variants "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-shopify-stores "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-shopify-stores) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-shopify-stores) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-shopify-stores) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-shopify-stores) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-shopify-stores) The same `/products.json` request returns a clean product catalog on one store and an HTML page on another. `www.allbirds.com` and `www.deathwishcoffee.com` return real `products` arrays on that route, while `www.apple.com`, which does not run on Shopify, answers with an HTML "Page Not Found" page and HTTP 404. A Shopify scraper needs a response classifier that tells a usable catalog apart from a rate limit, a blocked route, or a 200 that carries HTML instead of JSON. This guide covers scraping Shopify products, prices, variants, collections, and product-page structured data with Python, using the code from Scrapfly's open-source Shopify scraper. [**Shopify Scraper**github.com/scrapfly/scrapfly-scrapers/tree/main/shopify-scraper](https://github.com/scrapfly/scrapfly-scrapers/tree/main/shopify-scraper) ## Key Takeaways A Shopify scraper should test structured catalog data first, validate the shape of the response, and keep product-page JSON-LD as the fallback for stores where that route fails. - Preflight `/products.json` with a small `limit` and classify the response into one of six outcomes before committing to a full crawl - Normalize each product and its nested variants separately, since price, SKU, and availability live on the variant, not the parent product - Paginate with a bounded retry policy that backs off on HTTP 429 and logs an explicit stop reason, so a partial run stays distinguishable from a complete one - Parse product-page JSON-LD for both `Product` and `ProductGroup` schema types, and add `render_js=True` only for a page confirmed to lack both blocks Endpoint support is store-dependent, so treat every rule below as something to verify per store. For crawling a full catalog at scale, see [How to Scrape an Entire Product Catalogue with PythonLearn how to discover, crawl, and extract every product from an e-commerce catalog in Python, then keep that data fresh at scale.](https://scrapfly.io/blog/posts/how-to-scrape-large-product-catalogs) **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## What Data Can a Shopify Storefront Expose? A cooperative Shopify storefront exposes product identity, descriptions, tags, images, and a nested list of variants through structured JSON. One record from `www.allbirds.com/products.json` carries: - Product id `7199699927120` - Handle `mens-strider-medium-grey` - Title `Men's Strider - Medium Grey (Blizzard Sole)` - A separate price, SKU, and stock flag per variant The product is the parent record. Each variant carries its own id, option list, SKU, price, optional compare-at price, and availability flag. ### Which Shopify Product and Variant Fields Matter? The canonical scraper's `ShopifyProduct` and `ShopifyVariant` records prioritize the fields below over other keys a raw product carries. | Level | Field | Type | Nullable | Normalization rule | |---|---|---|---|---| | Product | `product_id` | string | No | Cast from the raw integer `id`, kept as a string for stable joins | | Product | `handle` | string | No | Used as-is to build `url` as `{store_root}/products/{handle}` | | Product | `tags` | list of strings | No | Kept raw, some themes mix merchandising tags with internal codes like `allbirds::hue => grey` | | Product | `options` | dict | No | Maps an option name like `Size` to its list of possible values | | Product | `description` | string | Yes | Flattened from `body_html` to plain text with a Selector's `string()` | | Product | `price_min` / `price_max` | float | Yes | Computed from the parsed variant prices, not read from a raw field | | Variant | `variant_id` | string | No | Cast from the raw integer `id` | | Variant | `sku` | string | Yes | Kept as `None` rather than an empty string when absent | | Variant | `price` / `compare_at_price` | float | Yes | Parsed from Shopify's price strings, for example `"91.00"`, with a helper that returns `None` on a bad value | | Variant | `available` | boolean | Yes | Kept as-is | ### How Do Shopify Products, Variants, and Collections Relate? One product record holds many variants nested inside it, so a men's shoe with thirteen sizes is one `product_id` with thirteen `variant_id` values, each with its own price and stock status. A `/collections//products.json` route returns the same product shape as `/products.json`, variants, images, tags, and all. Requesting `www.deathwishcoffee.com/collections/coffee/products.json` returns full records like `pumpkin-chai` with three sized variants (`1 bag`, `2 bags`, `5 bags`) at `13.99`, `27.98`, and `69.95`. With the record shape defined, the next step is confirming a target actually exposes it before writing a full crawler around it. ## How Do You Confirm a Store Uses Shopify? A valid Shopify-shaped response is a strong capability signal, but a failed route does not prove the site is not on Shopify. The only reliable test is a small, direct request to the catalog route itself, classified by status code, JSON validity, and the presence of a `products` array, not guessed from the domain or the theme. The code below needs Python 3.10 or newer, `pip install scrapfly-sdk parsel loguru`, and your API key in a `SCRAPFLY_KEY` environment variable. text ```text GET /products.json?limit=1&page=1 │ ├─ HTTP 429 → rate_limited ├─ HTTP non-2xx (403, 404, 5xx...) → unavailable ├─ HTTP 2xx, body is not valid JSON → not_json ├─ HTTP 2xx, JSON has no "products" → unexpected_json ├─ HTTP 2xx, "products" is empty → empty_catalog └─ HTTP 2xx, "products" is populated → shopify_catalog ``` Preflighting three stores gives two outcomes. `www.allbirds.com` and `www.deathwishcoffee.com` both return `shopify_catalog` with `content-type: application/json`. `www.apple.com` returns HTTP 404 with `content-type: text/html`, which classifies as `unavailable`. python ```python import json import os from typing import Dict, List, Optional, Tuple, TypedDict from urllib.parse import urljoin, urlparse from loguru import logger as log from scrapfly import ScrapeApiResponse, ScrapeConfig, ScrapflyClient SCRAPFLY = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"]) BASE_CONFIG = { "asp": True, "country": "US", } # the storefront JSON routes are classified by this scraper instead of failing on the status # code, so upstream 403/404/429 responses have to reach the parser instead of raising CLASSIFY_CONFIG = {**BASE_CONFIG, "raise_on_upstream_error": False} class ShopifyPreflight(TypedDict): store_url: str catalog_url: str status_code: int content_type: Optional[str] outcome: str product_count: Optional[int] is_shopify_catalog: bool def _store_root(store_url: str) -> str: """normalize a store address to a scheme and host root without a trailing slash""" parsed = urlparse(store_url if "//" in store_url else f"https://{store_url}") if not parsed.netloc: raise ValueError(f"cannot resolve a store host from {store_url!r}") return f"{parsed.scheme or 'https'}://{parsed.netloc}" def _catalog_url(store_url: str, page: int, limit: int, collection: Optional[str] = None) -> str: path = f"/collections/{collection}/products.json" if collection else "/products.json" return f"{_store_root(store_url)}{path}?limit={limit}&page={page}" def classify_catalog_response(response: ScrapeApiResponse) -> Tuple[str, Optional[List[Dict]]]: """classify a storefront products.json response before any field is read Returns an outcome label and the product list when the response is a usable catalog: - shopify_catalog: JSON body with a non empty top level products array - empty_catalog: JSON body with an empty products array, for example past the last page - rate_limited: HTTP 429, the route exists but the request was throttled - unavailable: any other non 2xx status, the route is missing, gated or blocked - not_json: 2xx body that does not parse as JSON, such as an HTML or login page - unexpected_json: 2xx JSON body without a products array, so not a Shopify catalog """ if response.status_code == 429: return "rate_limited", None if response.status_code // 100 != 2: return "unavailable", None try: data = json.loads(response.content) except json.JSONDecodeError: return "not_json", None products = data.get("products") if isinstance(data, dict) else None if not isinstance(products, list): return "unexpected_json", None return ("shopify_catalog" if products else "empty_catalog"), products def parse_preflight(response: ScrapeApiResponse, store_url: str) -> ShopifyPreflight: """turn a products.json preflight response into a capability report""" outcome, products = classify_catalog_response(response) return { "store_url": _store_root(store_url), "catalog_url": response.context["url"], "status_code": response.status_code, "content_type": response.scrape_result.get("response_headers", {}).get("content-type"), "outcome": outcome, "product_count": len(products) if products is not None else None, "is_shopify_catalog": outcome == "shopify_catalog", } async def check_shopify_stores(store_urls: List[str]) -> List[ShopifyPreflight]: """preflight the products.json route of each store to see whether it serves catalog JSON A Shopify shaped response is strong evidence the route is usable right now. A failed response does not prove the store is not on Shopify, only that this route did not answer. """ by_catalog_url = {_catalog_url(url, page=1, limit=1): url for url in store_urls} to_scrape = [ScrapeConfig(url, **CLASSIFY_CONFIG) for url in by_catalog_url] reports = [] async for response in SCRAPFLY.concurrent_scrape(to_scrape): if not isinstance(response, ScrapeApiResponse): continue try: report = parse_preflight(response, by_catalog_url[response.context["url"]]) log.info("preflight {} -> {}", report["store_url"], report["outcome"]) reports.append(report) except Exception as e: log.error(f"failed to preflight store: {e}") log.success(f"preflighted {len(reports)} stores") return reports ``` `check_shopify_stores` batches every store through Scrapfly's `concurrent_scrape`, so a slow or blocked store never stalls the rest of the batch, and each report logs its own outcome independently. ### What Does a Shopify `/products.json` Preflight Prove? It proves the route returned a Shopify-shaped catalog response at the moment of the request. A store can disable the route, rate-limit it per session, or leave it open indefinitely, and any of those can change without notice. Re-run it at the start of every crawl. ### Which Shopify Signals Need a Second Check? An asset host like `cdn.shopify.com` appearing in image URLs (every Allbirds product photo resolves to `cdn.shopify.com/s/files/1/1104/4168/...`) is a useful confirming signal even when the catalog route itself is blocked on a different store. Treat a signal like this as evidence to combine with a direct preflight, not proof on its own, and never treat an ambiguous signal as grounds for trying to bypass a password or login gate. ## How Do You Scrape Shopify `/products.json` with Python? When `/products.json` returns valid catalog JSON, parse that structured response before touching any theme-specific HTML. The route accepts `limit` and `page` query parameters and the storefront's highest accepted page size is `250`. ### How Should a Shopify Scraper Classify the First Response? Every page including the first one passes through the same `classify_catalog_response` function shown above. A 200 that carries an HTML body, such as a login page, classifies as `not_json` and never reaches the parser. A 429 classifies as `rate_limited`, which `_scrape_catalog_page` below answers with a backoff and a bounded retry. ### How Should a Shopify Scraper Paginate Product JSON? The storefront route does not report a total page count, so the stop condition has to come from the response itself: a page shorter than the requested limit, a non-catalog outcome, or a configured `max_pages` ceiling. python ```python import asyncio CATALOG_PAGE_SIZE = 250 # highest page size the storefront products.json route accepts CATALOG_PAGE_DELAY = 1.0 # seconds between catalog pages, Shopify rate limits bursts CATALOG_MAX_RETRIES = 3 async def _scrape_catalog_page(url: str) -> Tuple[str, Optional[List[Dict]]]: """fetch one catalog page, retrying a bounded number of times on a rate limit""" for attempt in range(CATALOG_MAX_RETRIES + 1): response = await SCRAPFLY.async_scrape(ScrapeConfig(url, **CLASSIFY_CONFIG)) outcome, products = classify_catalog_response(response) if outcome != "rate_limited" or attempt == CATALOG_MAX_RETRIES: return outcome, products backoff = CATALOG_PAGE_DELAY * 2 ** (attempt + 1) log.warning(f"rate limited on {url}, retrying in {backoff}s") await asyncio.sleep(backoff) return "rate_limited", None async def scrape_catalog( store_url: str, max_pages: int = 2, limit: int = CATALOG_PAGE_SIZE, collection: Optional[str] = None, ) -> List[ShopifyProduct]: """scrape products from a Shopify storefront products.json route Pagination is sequential and paced because the payload reports no page count, so the stop condition is a short, empty or failed page. Every run logs why it stopped, which keeps a partial run distinguishable from a complete one. """ store_root = _store_root(store_url) products: List[ShopifyProduct] = [] seen = set() stop_reason = f"reached max_pages={max_pages}" for page in range(1, max_pages + 1): url = _catalog_url(store_root, page=page, limit=limit, collection=collection) log.info(f"scraping catalog page {page} of {store_root}") outcome, page_products = await _scrape_catalog_page(url) if outcome != "shopify_catalog": stop_reason = f"page {page} returned {outcome}" break for product in page_products: if not product.get("id") or not product.get("handle") or product["id"] in seen: continue seen.add(product["id"]) products.append(parse_catalog_product(product, store_root)) if len(page_products) < limit: stop_reason = f"page {page} returned {len(page_products)} of {limit} products, last page" break await asyncio.sleep(CATALOG_PAGE_DELAY) log.success(f"scraped {len(products)} products from {store_root}, stopped: {stop_reason}") return products ``` `scrape_catalog` calls `parse_catalog_product`, the normalizer covered next, and returns its `ShopifyProduct` type. Both are defined in the next block. When assembling one file, put that block above this one: on Python 3.10 to 3.13 the `List[ShopifyProduct]` return annotation is evaluated at definition time and raises `NameError` if the type does not exist yet. The repository's `shopify.py` already uses that order. ### How Do You Normalize Shopify Product Variants? Preserve each variant's source id and parsed price separately, rather than flattening a product down to one price. Price and compare-at price are parsed with a small helper, since Shopify serializes them as strings like `"91.00"`. python ```python from parsel import Selector class ShopifyVariant(TypedDict): variant_id: str title: Optional[str] sku: Optional[str] options: List[str] price: Optional[float] compare_at_price: Optional[float] available: Optional[bool] class ShopifyProduct(TypedDict): product_id: str handle: str url: str title: Optional[str] vendor: Optional[str] product_type: Optional[str] description: Optional[str] tags: List[str] options: Dict[str, List[str]] images: List[str] variants: List[ShopifyVariant] price_min: Optional[float] price_max: Optional[float] published_at: Optional[str] updated_at: Optional[str] def _float(value) -> Optional[float]: """Shopify serializes storefront prices as strings, for example "91.00" """ try: return float(value) except (TypeError, ValueError): return None def _text(html: Optional[str]) -> Optional[str]: """flatten a Shopify body_html field into plain text""" if not html: return None return " ".join(Selector(text=html).xpath("string()").get("").split()) or None def parse_variant(variant: Dict) -> ShopifyVariant: """parse a single nested variant of a Shopify catalog product""" return { "variant_id": str(variant["id"]), "title": variant.get("title"), "sku": variant.get("sku") or None, "options": [variant[key] for key in ("option1", "option2", "option3") if variant.get(key)], "price": _float(variant.get("price")), "compare_at_price": _float(variant.get("compare_at_price")), "available": variant.get("available"), } def parse_catalog_product(product: Dict, store_root: str) -> ShopifyProduct: """parse one product of a Shopify catalog response, keeping every variant intact""" variants = [parse_variant(v) for v in product.get("variants") or [] if v.get("id")] prices = [v["price"] for v in variants if v["price"] is not None] return { "product_id": str(product["id"]), "handle": product["handle"], "url": f"{store_root}/products/{product['handle']}", "title": product.get("title"), "vendor": product.get("vendor"), "product_type": product.get("product_type") or None, "description": _text(product.get("body_html")), "tags": product.get("tags") or [], "options": {o["name"]: o.get("values") or [] for o in product.get("options") or [] if o.get("name")}, "images": [i["src"] for i in product.get("images") or [] if i.get("src")], "variants": variants, "price_min": min(prices) if prices else None, "price_max": max(prices) if prices else None, "published_at": product.get("published_at"), "updated_at": product.get("updated_at"), } ``` A men's shoe with thirteen sizes stays one `product_id` with thirteen distinct `variant_id` entries, each keeping its own SKU, price, and availability, so a sold-out size never silently disappears into an average. Catalog data covers most of what a product listing needs. Category context and full-site discovery live one layer up, in collections and sitemaps. ## How Do You Scrape Shopify Collections and Product URLs? Use Shopify collections to preserve merchandising context and use sitemap discovery when the catalog JSON route cannot enumerate the full product set on its own. Collections share the exact same route shape as the main catalog, just scoped under `/collections//products.json`, so `scrape_catalog` above already handles them through its `collection` parameter. python ```python async def scrape_collection( store_url: str, collection: str, max_pages: int = 2, limit: int = CATALOG_PAGE_SIZE ) -> List[ShopifyProduct]: """scrape products of a single Shopify collection, keeping the merchandising context Collection JSON support is store dependent: a store that serves /products.json can still return an empty list here, so treat an empty result as an unusable route, not an empty store. """ return await scrape_catalog(store_url, max_pages=max_pages, limit=limit, collection=collection) ``` Against `www.deathwishcoffee.com` the `coffee` collection returns full product records, `pumpkin-chai` among them, in the same shape the main catalog route returns. Discovering collection handles themselves through `/collections.json` or the site's own navigation sits outside what this function does. ### When Should a Shopify Scraper Use Collection Pages? Reach for a collection when the category a product belongs to matters for the output, not just the product data itself, and when the target store's collection route is confirmed to return real data instead of an empty or blocked response. ### When Should a Shopify Scraper Use an XML Sitemap? Reach for the sitemap when catalog JSON enumeration is unavailable or clearly incomplete. Shopify's `/sitemap.xml` is a sitemap index, not a product sitemap, so read it first. Requesting a numbered product sitemap path directly returns HTTP 400. python ```python def parse_sitemap_locations(response: ScrapeApiResponse) -> List[str]: """parse every loc entry of a sitemap index or a sitemap""" # sitemaps are namespaced XML, and parsel 1.12+ parses them as XML (it detects the entries selector = Selector(text=response.content, type="xml") namespaces = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"} return selector.xpath("//sm:loc/text()", namespaces=namespaces).getall() async def scrape_product_urls(store_url: str, max_sitemaps: int = 1) -> List[str]: """discover product URLs from the Shopify sitemap index Shopify links its product sitemaps from /sitemap.xml with an id range query, so the index has to be read first: the numbered sitemap path on its own answers with a 400. """ store_root = _store_root(store_url) index = await SCRAPFLY.async_scrape(ScrapeConfig(f"{store_root}/sitemap.xml", **BASE_CONFIG)) product_sitemaps = [loc for loc in parse_sitemap_locations(index) if "sitemap_products" in loc][:max_sitemaps] log.info(f"found {len(product_sitemaps)} product sitemaps on {store_root}") urls = [] to_scrape = [ScrapeConfig(loc, **BASE_CONFIG) for loc in product_sitemaps] async for response in SCRAPFLY.concurrent_scrape(to_scrape): if not isinstance(response, ScrapeApiResponse): continue try: urls.extend(urljoin(store_root, loc) for loc in parse_sitemap_locations(response) if "/products/" in loc) except Exception as e: log.error(f"failed to scrape product sitemap: {e}") log.success(f"scraped {len(urls)} product urls from {store_root}") return urls ``` Scrapfly #### Scale your web scraping effortlessly Scrapfly handles proxies, browsers, and anti-bot bypass — so you can focus on data. [Try Free →](https://scrapfly.io/register)A single product sitemap from `www.allbirds.com` returned 692 product URLs, each pointing at a real `/products/` path ready for the JSON-LD parser in the next section. Collections and sitemaps assume the catalog route is reachable in the first place. The next section covers what to do when it is not. ## What Should a Shopify Scraper Do When `/products.json` Fails? When the catalog route fails, the classified outcome picks the scraper's next step, so a blocked store never ends as a silent empty result. | Outcome | Observed evidence | Next action | |---|---|---| | `not_json` | A 2xx response whose body does not parse as JSON, such as an HTML or login page | Reject the response as a semantic mismatch regardless of the 2xx status | | `rate_limited` | HTTP 429 on a route that answered normally moments earlier | Retry with bounded exponential backoff, `_scrape_catalog_page` doubles its delay per attempt up to `CATALOG_MAX_RETRIES` | | `unavailable` | Any non-2xx status such as 403 or 404 on the catalog route, for example `www.apple.com/products.json`, which returns HTTP 404 with an HTML page | Treat the route as protected or missing for this store, switch to the product-page fallback, do not assume a root cause | | `empty_catalog` | A valid JSON body with an empty `products` array | Confirm against another signal, such as a known collection or sitemap entry, before assuming the store has zero products | | `unexpected_json` | A 2xx JSON body without a top-level `products` list, such as an error payload | Reject the response, the route exists but is not the storefront catalog shape expected | When `/products.json` is out, the next place to look is the product page's own embedded structured data. python ```python class ShopifyOffer(TypedDict): sku: Optional[str] title: Optional[str] price: Optional[float] currency: Optional[str] availability: Optional[str] url: Optional[str] class ShopifyProductPage(TypedDict): url: str schema_type: Optional[str] product_id: Optional[str] name: Optional[str] brand: Optional[str] sku: Optional[str] description: Optional[str] images: List[str] price: Optional[float] currency: Optional[str] availability: Optional[str] offers: List[ShopifyOffer] variant_urls: List[str] def _ld_nodes(response: ScrapeApiResponse) -> List[Dict]: """collect every JSON-LD node of a page, flattening arrays and @graph containers""" nodes = [] for raw in response.selector.css('script[type="application/ld+json"]::text').getall(): try: data = json.loads(raw) except json.JSONDecodeError: log.warning("skipping invalid JSON-LD block on {}", response.context["url"]) continue for node in data if isinstance(data, list) else [data]: if not isinstance(node, dict): continue graph = node.get("@graph") if isinstance(graph, list): nodes.extend(g for g in graph if isinstance(g, dict)) else: nodes.append(node) return nodes def _parse_offers(offers) -> List[ShopifyOffer]: """Shopify themes emit offers as a single object or as a list of variant offers""" entries = offers if isinstance(offers, list) else [offers] if isinstance(offers, dict) else [] return [ { "sku": offer.get("sku"), "title": offer.get("name"), "price": _float(offer.get("price") or (offer.get("priceSpecification") or {}).get("price")), "currency": offer.get("priceCurrency"), "availability": offer.get("availability"), "url": offer.get("url"), } for offer in entries if isinstance(offer, dict) ] def parse_product_page(response: ScrapeApiResponse) -> ShopifyProductPage: """parse a Shopify product page through its JSON-LD block Themes use either Product or ProductGroup, so the schema type is validated before any field is mapped, and the type stays on the record. """ node = next((n for n in _ld_nodes(response) if n.get("@type") in ("Product", "ProductGroup")), None) if node is None: raise ValueError(f"no Product or ProductGroup JSON-LD on {response.context['url']}") offers = _parse_offers(node.get("offers")) brand = node.get("brand") images = node.get("image") or [] return { "url": response.context["url"], "schema_type": node.get("@type"), "product_id": node.get("productGroupID") or node.get("@id"), "name": node.get("name"), "brand": brand.get("name") if isinstance(brand, dict) else brand, "sku": node.get("sku"), "description": " ".join((node.get("description") or "").split()) or None, "images": [i for i in (images if isinstance(images, list) else [images]) if isinstance(i, str)], "price": offers[0]["price"] if offers else None, "currency": offers[0]["currency"] if offers else None, "availability": offers[0]["availability"] if offers else None, "offers": offers, "variant_urls": [v["url"] for v in node.get("hasVariant") or [] if isinstance(v, dict) and v.get("url")], } ``` Two real pages return two different shapes. Death Wish Coffee's `vanilla-10oz` embeds a `Product` block, priced at `11.99` and `InStock`. Allbirds' `mens-strider-medium-grey` embeds a `ProductGroup`, priced at `130` and `InStock` at the group level, with 91 `variant_urls` extracted from 104 `hasVariant` entries. ### How Do 403, 404, and 429 Responses Change the Shopify Fallback? A 429 means retry the same route later. A 403 or 404 means switch to the product-page fallback for that store rather than retrying the JSON route indefinitely. Neither status alone tells you whether the block is permanent, temporary, or store policy, so log the classification and move on instead of guessing at the cause. ### How Do You Parse Shopify Product JSON-LD? Check the `@type` before trusting any field, since Shopify themes emit both `Product` and `ProductGroup` depending on the theme. Unwrap `@graph` containers and lists before filtering, since not every theme puts the product block at the top level of the JSON-LD payload, and prefer this structured data over a raw CSS selector on the page's `

`, since a theme can render that heading blank in static HTML and fill it in client-side. ### When Does a Shopify Product Page Need JavaScript Rendering? Render only after both the JSON route and the embedded JSON-LD come up empty. The scraper's default config, `asp=True` and `country="US"`, doesn't enable rendering, since JSON-LD alone answered every page tested. Add `render_js=True` only once a specific page is confirmed missing its `Product` or `ProductGroup` block. JSON-LD answered both product pages tested here without rendering. ## How Do You Scrape Shopify Stores Reliably with Scrapfly? Both test stores served `/products.json` to a plain HTTP client without a challenge, so the catalog route itself does not require Scrapfly. The friction showed up with volume: after a handful of direct requests, Allbirds and Death Wish Coffee both answered product pages with HTTP 429 and `retry-after: 60`. Sending every request through one `ScrapeConfig` shape keeps the same classifier and parser for those stores and for stores that do block plain requests, with transport retries and optional rendering configured in one place. python ```python async def scrape_product_pages(urls: List[str]) -> List[ShopifyProductPage]: """scrape Shopify product pages, the fallback when the catalog JSON route is unusable""" to_scrape = [ScrapeConfig(url, **BASE_CONFIG) for url in urls] products = [] async for response in SCRAPFLY.concurrent_scrape(to_scrape): if not isinstance(response, ScrapeApiResponse): continue try: log.info("scraping product page {}", response.context["url"]) products.append(parse_product_page(response)) except Exception as e: log.error(f"failed to scrape product page: {e}") log.success(f"scraped {len(products)} product pages") return products ``` `scrape_product_pages` reuses the same `BASE_CONFIG` shown earlier and batches every URL through `concurrent_scrape` exactly like `check_shopify_stores` does for stores and `scrape_product_urls` does for sitemaps. ### Which Scrapfly Request Options Should the Shopify Scraper Enable? `asp=True` turns on Scrapfly's [Web Unblocker](https://scrapfly.io/products/unblocker) (previously ASP; SDK 0.12.0 and newer also accept `unblocker=True`) for stores that block direct requests. Keep `raise_on_upstream_error=False` on the JSON routes so a 403 or 429 reaches the classifier instead of raising. Add `render_js=True` only when a confirmed page has no usable JSON-LD. ### How Should a Shopify Scraper Pace Retries Across Stores? Keep retries and page delay bounded per store, since a rate limit applies to that store's session, not the whole scraper. Batch independent requests through `concurrent_scrape` but keep a single store's paginated catalog sequential so its backoff applies correctly. Report partial runs with the same `stop_reason` pattern from pagination. Scrapfly also ships [Python](https://scrapfly.io/docs/sdk/python), TypeScript, Go, and Rust SDKs. ## FAQ Does Every Shopify Store Expose `/products.json`?No. Treat it as a capability to test, not a guarantee. A failed response should trigger product-page or discovery fallbacks rather than a claim that the target is not Shopify. Do Shopify Product Variants Have Different Prices?They can. Preserve each variant as its own nested record so price, comparison price, SKU, options, and availability are not incorrectly collapsed to the product level. Does Public Shopify Storefront Scraping Require an Admin API Token?Not for public pages and routes that the storefront exposes without authentication. Keep this distinct from Shopify Admin APIs, which are authenticated and have separate documentation and pagination behavior. Should a Shopify Scraper Render JavaScript for Every Product Page?No. Try structured endpoint data and embedded JSON-LD first. Render only when the required field is absent from static responses and a controlled test confirms it appears after rendering. ## Conclusion: Preflight Every Shopify Store Before Scraping It The sequence holds across every store this guide tested. Preflight the catalog route. Classify the response into one of six outcomes. Parse products and variants when the JSON is there. Paginate with bounded retries. Fall back to JSON-LD, and render only when a confirmed gap requires it. One scraper is reusable across storefronts, but the same request returned catalog JSON from Allbirds and Death Wish Coffee and a 404 page from Apple, so every new store still needs its own preflight. [Scrapfly](https://scrapfly.io/products/web-scraping-api) carries that same classifier and parser through the stores that block direct requests, with `render_js` enabled only when needed. ### Web Scraping API Scrape any website with our powerful API. Anti-bot bypass, JavaScript rendering, and rotating proxies built-in. [Try Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started) Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents - [Key Takeaways](#key-takeaways) - [What Data Can a Shopify Storefront Expose?](#what-data-can-a-shopify-storefront-expose) - [Which Shopify Product and Variant Fields Matter?](#which-shopify-product-and-variant-fields-matter) - [How Do Shopify Products, Variants, and Collections Relate?](#how-do-shopify-products-variants-and-collections-relate) - [How Do You Confirm a Store Uses Shopify?](#how-do-you-confirm-a-store-uses-shopify) - [What Does a Shopify /products.json Preflight Prove?](#what-does-a-shopify-products-json-preflight-prove) - [Which Shopify Signals Need a Second Check?](#which-shopify-signals-need-a-second-check) - [How Do You Scrape Shopify /products.json with Python?](#how-do-you-scrape-shopify-products-json-with-python) - [How Should a Shopify Scraper Classify the First Response?](#how-should-a-shopify-scraper-classify-the-first-response) - [How Should a Shopify Scraper Paginate Product JSON?](#how-should-a-shopify-scraper-paginate-product-json) - [How Do You Normalize Shopify Product Variants?](#how-do-you-normalize-shopify-product-variants) - [How Do You Scrape Shopify Collections and Product URLs?](#how-do-you-scrape-shopify-collections-and-product-urls) - [When Should a Shopify Scraper Use Collection Pages?](#when-should-a-shopify-scraper-use-collection-pages) - [When Should a Shopify Scraper Use an XML Sitemap?](#when-should-a-shopify-scraper-use-an-xml-sitemap) - [What Should a Shopify Scraper Do When /products.json Fails?](#what-should-a-shopify-scraper-do-when-products-json-fails) - [How Do 403, 404, and 429 Responses Change the Shopify Fallback?](#how-do-403-404-and-429-responses-change-the-shopify-fallback) - [How Do You Parse Shopify Product JSON-LD?](#how-do-you-parse-shopify-product-json-ld) - [When Does a Shopify Product Page Need JavaScript Rendering?](#when-does-a-shopify-product-page-need-javascript-rendering) - [How Do You Scrape Shopify Stores Reliably with Scrapfly?](#how-do-you-scrape-shopify-stores-reliably-with-scrapfly) - [Which Scrapfly Request Options Should the Shopify Scraper Enable?](#which-scrapfly-request-options-should-the-shopify-scraper-enable) - [How Should a Shopify Scraper Pace Retries Across Stores?](#how-should-a-shopify-scraper-pace-retries-across-stores) - [FAQ](#faq) - [Conclusion: Preflight Every Shopify Store Before Scraping It](#conclusion-preflight-every-shopify-store-before-scraping-it) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ python scrapeguide ### How to Scrape Lowe's Product, Price, Search, and Store Location Data Scrape Lowe's product and pricing data from embedded page state, discover products and stores, and handle Akamai with Sc... ](https://scrapfly.io/blog/posts/how-to-scrape-lowe) [ python data-parsing ### How to Scrape idealo: Product Offers, Prices, and Price History Scrape idealo product pages for the full merchant offer list, prices, and price history with Python and the Scrapfly Web... ](https://scrapfly.io/blog/posts/how-to-scrape-idealo) [ python scrapeguide ### How to Scrape Amazon.com Product Data and Reviews This scrape guide covers the biggest e-commerce platform in US - Amazon.com. We'll take a look how to scrape product dat... ](https://scrapfly.io/blog/posts/how-to-scrape-amazon) Scale your web scraping effortlessly, **1,000 free credits** [Start Free](https://scrapfly.io/register)