[Blog](https://scrapfly.io/blog) / [hidden-api](https://scrapfly.io/blog/tag/hidden-api) / [How to Scrape Apple App Store Reviews and App Data](https://scrapfly.io/blog/posts/how-to-scrape-app-store) # How to Scrape Apple App Store Reviews and App Data by [Hisham Medhat](https://scrapfly.io/blog/author/hisham) Nov 10, 2026 24 min read [\#hidden-api](https://scrapfly.io/blog/tag/hidden-api) [\#python](https://scrapfly.io/blog/tag/python) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-app-store "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-app-store&text=How%20to%20Scrape%20Apple%20App%20Store%20Reviews%20and%20App%20Data "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-app-store "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-app-store) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-app-store) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-app-store) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-app-store) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-app-store) The Apple App Store is mostly known for its app listings, though it contains much more public data like ratings, the star distribution and customer reviews, most of it in Apple's own JSON endpoints. Take ChatGPT, for example - a single iTunes Lookup request gave us 37 of its metadata fields, the RSS feed gave us 500 reviews and `userReviewsRow` reached reviews from November 2024. These are the three endpoints we'll use to scrape apps.apple.com, in Python, starting from Scrapfly's maintained App Store scraper and adding the review code it doesn't include yet, so let's dive in! [How to Scrape Google Play App Reviews and DataScrape Google Play app metadata, ratings, and the full review set with Python, past the few-hundred-review ceiling the free library hits.](https://scrapfly.io/blog/posts/how-to-scrape-google-play-app-reviews-and-data) [**Latest App Store Scraper Code**github.com/scrapfly/scrapfly-scrapers/tree/main/app-store-scraper](https://github.com/scrapfly/scrapfly-scrapers/tree/main/app-store-scraper) ## Key Takeaways - One iTunes Lookup request gave us 37 metadata fields for ChatGPT. - RSS page 11 got HTTP 400 with `CustomerReviews RSS page depth is limited to 10`. - `userReviewsRow` went much deeper: on November 3, 2026, index 130,000 still returned 100 rows, all from November 2024! - Unknown app IDs came back as HTTP 200 with `resultCount: 0`. - On November 3, 2026, RSS times still ended in `-07:00` after daylight saving time ended, so we keep the `Z` time when a review is in both. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## Project Setup We'll be using Python with four packages: `scrapfly-sdk` to fetch pages, [parsel](https://scrapfly.io/blog/posts/guide-to-html-parsing-with-parsel-python) to parse them, `loguru` for logging and [httpx](https://scrapfly.io/blog/posts/web-scraping-with-python-httpx) for the deep review endpoint. These can be installed via `pip`: bash ```bash pip install scrapfly-sdk parsel loguru httpx export SCRAPFLY_KEY="YOUR_SCRAPFLY_KEY" ``` Then, we'll create a single Scrapfly client and a base config, which every request in our scraper shares. Here, `unblocker` enables Scrapfly's Unblocker (previously ASP), while `country` pins our exit IP to the US: python ```python import json import os import re from typing import Dict, List, Optional from loguru import logger as log from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse, ScrapflyError SCRAPFLY = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"]) BASE_CONFIG = { "unblocker": True, "country": "US", } ``` Note that the deep review section is the one exception, as it calls Apple with plain `httpx` instead of this client. ## What Apple App Store Data Can You Scrape? Here's what each of Apple's six sources gave us for ChatGPT: | Source | Authentication | What we got for ChatGPT | Limit we hit | |---|---|---|---| | iTunes Lookup API | None | 37 mapped fields, from `bundleId` to `supportedDevices` | `resultCount: 0` for unknown IDs, empty `screenshots` | | RSS review feed | None | 500 newest reviews over 10 pages | Page 11 gets HTTP 400, empty pages under parallel calls | | customer-reviews summary | Store-front header | 133,762 US reviews, 5-bucket star spread | Header picks the storefront | | `userReviewsRow` | Store-front header | Rows at index 130,000 | HTML error page without the header | | [App Store Connect API](https://developer.apple.com/app-store-connect/api/) | JWT, app owner | Not tested here | Owned apps only | | `apps.apple.com` page | Scrapfly key | JSON-LD with 10.8M ratings | Subset of Lookup fields | [Web Scraping Background Requests with Headless BrowsersIn this tutorial we'll be taking a look at a rather new and popular web scraping technique - capturing background requests using headless browsers.](https://scrapfly.io/blog/posts/web-scraping-background-requests-with-headless-browsers-and-python) ## How Do You Find an Apple App Store App ID and Country Storefront? To scrape App Store data all we need is an app ID and a country storefront, and the app URL already holds both of these values. For example, the ChatGPT app at `https://apps.apple.com/us/app/chatgpt/id6448311069` has its country (`us`) in the two-letter segment before `/app/` and its app ID (`6448311069`) in the digits after `id`. The `chatgpt` slug in between is there for human readers, as none of Apple's endpoints ask for it - they key on `6448311069` alone. Meaning, our scraper keeps working even if the app gets renamed! Our scraper's first function, `parse_app_url`, reads both values out of a full app URL and also takes a bare numeric ID on its own: python ```python def parse_app_url(value: str) -> tuple[str, str]: """Return (app_id, country) from an App Store URL or a bare numeric app ID.""" if value.isdigit(): return value, "us" match = re.search(r"apps\.apple\.com/([a-z]{2})/app/[^/]+/id(\d+)", value) if not match: raise ValueError(f"not an App Store app URL: {value}") return match.group(2), match.group(1) print(parse_app_url("https://apps.apple.com/us/app/chatgpt/id6448311069")) print(parse_app_url("6448311069")) ``` python ```python ('6448311069', 'us') ('6448311069', 'us') ``` Both calls printed `('6448311069', 'us')`, though only the first one read `us` from the URL - for the bare ID, our helper returns `us` itself, the same default as the scraper's `country="us"`. Further, any value that isn't all digits and doesn't match our regular expression gets a `ValueError` instead. Both values go into every request from here on, so let's take a look at how we can use them to retrieve the app metadata first. ## How Do You Scrape App Metadata With the iTunes Lookup API? To scrape app metadata we can use the iTunes Lookup endpoint - `https://itunes.apple.com/lookup`, which takes the app's `id` and a `country` code for the storefront. Both of these go straight into the query string of our `lookup_url`: python ```python def parse_app_metadata(response: ScrapeApiResponse, app_id: str, country: str) -> Dict: """parse app metadata from the iTunes lookup API""" data = json.loads(response.content) if not data.get("resultCount"): raise ValueError(f"App {app_id} not found in {country} storefront") app = data["results"][0] price = app.get("price") or 0 return { "appId": str(app.get("trackId", app_id)), "bundleId": app.get("bundleId"), "title": app.get("trackName"), "description": app.get("description"), "summary": app.get("releaseNotes"), "developer": app.get("artistName"), "developerId": app.get("artistId"), "developerUrl": app.get("artistViewUrl"), "seller": app.get("sellerName"), "sellerUrl": app.get("sellerUrl"), "genre": app.get("primaryGenreName"), "genres": app.get("genres", []), "genreIds": app.get("genreIds", []), "price": price, "free": price == 0, "currency": app.get("currency"), "formattedPrice": app.get("formattedPrice"), "score": app.get("averageUserRating"), "scoreCurrentVersion": app.get("averageUserRatingForCurrentVersion"), "ratings": app.get("userRatingCount"), "ratingsCurrentVersion": app.get("userRatingCountForCurrentVersion"), "version": app.get("version"), "released": app.get("releaseDate"), "updated": app.get("currentVersionReleaseDate"), "contentRating": app.get("contentAdvisoryRating") or app.get("trackContentRating"), "minimumOsVersion": app.get("minimumOsVersion"), "fileSizeBytes": app.get("fileSizeBytes"), "languages": app.get("languageCodesISO2A", []), "icon": app.get("artworkUrl512") or app.get("artworkUrl100"), "screenshots": app.get("screenshotUrls", []), "ipadScreenshots": app.get("ipadScreenshotUrls", []), "appletvScreenshots": app.get("appletvScreenshotUrls", []), "supportedDevices": app.get("supportedDevices", []), "advisories": app.get("advisories", []), "features": app.get("features", []), "url": app.get("trackViewUrl"), "country": country.lower(), } async def scrape_app_metadata(app_id: str, country: str = "us") -> Dict: """scrape app metadata via the iTunes lookup API, falling back to the apps.apple.com page""" lookup_url = f"https://itunes.apple.com/lookup?id={app_id}&country={country}" response = await SCRAPFLY.async_scrape(ScrapeConfig(lookup_url, **BASE_CONFIG)) try: metadata = parse_app_metadata(response, app_id, country) except: log.info("no iTunes data for {}, falling back to apps.apple.com", app_id) page_url = f"https://apps.apple.com/{country}/app/id{app_id}" response = await SCRAPFLY.async_scrape(ScrapeConfig(page_url, **BASE_CONFIG)) metadata = parse_app_metadata_from_page(response, app_id, country) log.success("scraped metadata for {} ({})", metadata["title"], app_id) return metadata ``` In our code above, we use `resultCount` to check whether Lookup found the app, raising a `ValueError` when it's missing or zero. Otherwise, we load the app from `results[0]` and parse out a 37-key record, renaming a few keys like `trackName` to `title` and `averageUserRating` to `score`. Further, since an unknown app ID is what triggers that `ValueError`, `scrape_app_metadata` catches it and requests `page_url` instead, which points at the app's page on apps.apple.com. Our `except` doesn't name an exception type though, so anything raised inside `parse_app_metadata` ends up in that same branch. We'll take a closer look at the page parser, `parse_app_metadata_from_page`, in the app page section below. In the scraper's sample run, ChatGPT came out with 37 fields, and here are 18 of them: json ```json { "appId": "6448311069", "bundleId": "com.openai.chat", "title": "ChatGPT", "developer": "OpenAI OpCo, LLC", "genre": "Productivity", "price": 0, "free": true, "currency": "USD", "score": 4.83076, "ratings": 8977205, "version": "1.2026.202", "updated": "2026-07-27T16:41:51Z", "contentRating": "12+", "minimumOsVersion": "17.0", "fileSizeBytes": "205886464", "screenshots": [], "url": "https://apps.apple.com/us/app/chatgpt/id6448311069?uo=4", "country": "us" } ``` As we can see, two of these values come from our parser rather than from Apple, and the first one is `free`, which is `true` because `price` is 0. The other one is `screenshots`, which is `[]` since ChatGPT's US storefront response had no `screenshotUrls` - every other array in our parser falls back the same way. The `ratings` count from this run was 8977205. Two months later the same call returned 10,778,636, so we'll save a timestamp with every record! That's all the metadata we need, but what about reviews? They live behind a different endpoint, and which one we pick depends on how far back we need to go. ## How Do You Scrape Apple App Store Customer Reviews? To scrape reviews we'll take a look at two of Apple's endpoints. The newest reviews are in the RSS feed, 50 per page for up to 10 pages, while anything older we'll walk through `userReviewsRow` in index windows with a store-front header. ### How do you parse customer reviews from the RSS/JSON feed? Let's start with the RSS feed. If we look at one of its entries, we can see that every value is wrapped in a `{"label": ...}` object, so our scraper unwraps them with a small `_label` helper. Not every entry is a review though, and the ones without an `im:rating` key get skipped by our parser: python ```python def _label(entry: Dict, key: str) -> Optional[str]: """RSS feed values are wrapped like {"label": "..."}""" return (entry.get(key) or {}).get("label") def parse_reviews(response: ScrapeApiResponse) -> List[Dict]: """parse customer reviews from the iTunes RSS JSON feed""" data = json.loads(response.content) entries = data.get("feed", {}).get("entry") or [] if isinstance(entries, dict): entries = [entries] reviews = [] for entry in entries: if not entry.get("im:rating"): continue author = entry.get("author", {}) link = entry.get("link", {}).get("attributes", {}) reviews.append({ "reviewId": _label(entry, "id"), "userName": author.get("name", {}).get("label"), "userUrl": author.get("uri", {}).get("label"), "title": _label(entry, "title"), "content": _label(entry, "content"), "score": int(_label(entry, "im:rating") or 0), "version": _label(entry, "im:version"), "voteSum": int(_label(entry, "im:voteSum") or 0), "voteCount": int(_label(entry, "im:voteCount") or 0), "updated": _label(entry, "updated"), "url": link.get("href"), }) return reviews async def scrape_reviews(app_id: str, country: str = "us", max_pages: int = None) -> List[Dict]: """scrape app reviews via the iTunes customer reviews RSS feed""" pages = min(max_pages or 10, 10) to_scrape = [ ScrapeConfig( url=f"https://itunes.apple.com/{country}/rss/customerreviews/page={page}/id={app_id}/sortby=mostrecent/json", **BASE_CONFIG, ) for page in range(1, pages + 1) ] reviews = [] async for response in SCRAPFLY.concurrent_scrape(to_scrape): if isinstance(response, ScrapflyError): # a failed request comes back as the error object, not raised log.error(f"skipping a page that could not be scraped: {response}") continue reviews.extend(parse_reviews(response)) log.success("scraped {} reviews for {}", len(reviews), app_id) return reviews ``` Our scraper above fetches all of the feed pages in parallel through `concurrent_scrape` and passes each one to `parse_reviews`, skipping any request that failed outright. The `isinstance(entries, dict)` check is there for a feed page with exactly one review, as Apple sends an object instead of a list in that case. Which in the scraper's sample run gave us this as the first review: json ```json { "reviewId": "14367092261", "userName": "Pyschotic Gamer", "userUrl": "https://itunes.apple.com/us/reviews/id1137004902", "title": "Addicted", "content": "Worth the money", "score": 5, "version": "1.2026.202", "voteSum": 0, "voteCount": 0, "updated": "2026-07-30T09:13:05-07:00", "url": "https://itunes.apple.com/us/review?id=6448311069&type=Purple%20Software" } ``` The `version` field shows which app release the reviewer was using. Note that this is something the deep endpoint below doesn't return. ### How should you paginate and de-duplicate Apple App Store reviews? The RSS feed paginates through `page=1` to `page=10` in the URL, and our `scrape_reviews` caps `max_pages` at 10. Asking for page 11 got us HTTP 400 with `CustomerReviews RSS page depth is limited to 10`. So if we run a full scrape, we should get all 500 reviews? Not exactly. A full run of `scrape_reviews` collected 300 reviews, and nothing was logged as an error. To find out why, we printed what each page returned. Pages 1, 2, 5 and 7 came back as HTTP 200 with an 875-byte feed and no `entry` array. The other six held 50 reviews each, and that's our 300! To get around this we can fetch the pages one at a time instead, and retry any page that comes back empty: python ```python import asyncio async def scrape_reviews_sequential(app_id: str, country: str = "us", max_pages: int = 10) -> List[Dict]: """fetch RSS review pages one at a time and retry any page that comes back empty""" reviews = [] for page in range(1, min(max_pages, 10) + 1): url = f"https://itunes.apple.com/{country}/rss/customerreviews/page={page}/id={app_id}/sortby=mostrecent/json" batch = [] for attempt in range(4): response = await SCRAPFLY.async_scrape(ScrapeConfig(url, **BASE_CONFIG)) batch = parse_reviews(response) if batch: break await asyncio.sleep(5 * (attempt + 1)) reviews.extend(batch) return reviews ``` By fetching pages one by one, `scrape_reviews_sequential` collected all 500 reviews in one run, with 500 unique `reviewId` values! Note that `asyncio.sleep` waits 5, 10 and then 15 seconds between attempts. If the fourth try is empty too, there's one more 20-second sleep before the next page. With a shorter backoff of 1, 2 and 4 seconds, one page still came back empty. That being said, if an app has fewer than 500 reviews its last pages are empty for real, and each one of them burns four attempts. So, to avoid that waste we can cap `max_pages` from the app's review count. The only reviews we're missing now are the ones past 500, where the RSS feed runs out. For those, we'll have to switch over to the `userReviewsRow` endpoint, so let's take a look at how we can do that. ### How do you collect more than 500 Apple App Store reviews? The `userReviewsRow` endpoint takes a `startIndex` and `endIndex` window and expects an `X-Apple-Store-Front` header. It's this header that Apple actually checks - without it we get a 4,980-character HTML error page back, even though the status is HTTP 200. The same header also picks the storefront. For example, when we requested the `gb` summary URL with the US header, we got US totals back, meaning the country in the URL does nothing on these endpoints. So, to select the storefront when scraping we have to send each country its own code with a matching language suffix: python ```python import httpx STOREFRONTS = {"us": "143441-1", "gb": "143444-2", "de": "143443-4"} def apple_headers(country: str) -> dict: return {"X-Apple-Store-Front": f"{STOREFRONTS[country]},29"} def fetch_review_summary(app_id: str, country: str = "us") -> dict: response = httpx.get( f"https://itunes.apple.com/{country}/customer-reviews/id{app_id}", params={"dataOnly": "true", "displayable-kind": 11}, headers=apple_headers(country), timeout=20, follow_redirects=True, ) data = response.json() return { "country": country, "review_count": data["totalNumberOfReviews"], "rating_count": data["ratingCount"], "rating_breakdown": data["ratingCountList"], } for cc in ("us", "gb", "de"): print(fetch_review_summary("6448311069", cc)) ``` Here, `fetch_review_summary` sends the header from `apple_headers` for each of our three storefronts. Note that the suffix has to match as well - a mismatched one like `143444-1` for the UK gets HTTP 400 with an empty body instead. Before we start paging, the summary already shows us how deep each storefront goes: python ```python {'country': 'us', 'review_count': 133762, 'rating_count': 10783354, 'rating_breakdown': [242845, 60296, 130354, 444671, 9905188]} {'country': 'gb', 'review_count': 17096, 'rating_count': 1331222, 'rating_breakdown': [32075, 8623, 18855, 68315, 1203354]} {'country': 'de', 'review_count': 14013, 'rating_count': 1697848, 'rating_breakdown': [51414, 14553, 36081, 134270, 1461530]} ``` `ratingCountList` runs from one star to five stars. As you can see, the three storefronts differ by an order of magnitude, so we'll keep the country code on every record. The summary and rows calls use plain `httpx` instead of our Scrapfly client, as the header alone gets JSON back. Sending the same header through `SCRAPFLY` also returned JSON though, so if Apple starts blocking our IP we can switch over. Now that we know which header to send, let's take a look at the reviews themselves. Our rows fetcher walks through the list in windows of 100 and maps each row onto the same keys as `parse_reviews`: python ```python import time ROWS_URL = "https://itunes.apple.com/WebObjects/MZStore.woa/wa/userReviewsRow" PAGE_SIZE = 100 def fetch_review_rows(app_id: str, country: str, start: int, sort: int = 4) -> list | None: """one review window; returns None when the store answers with its HTML error page""" for attempt in range(3): response = httpx.get(ROWS_URL, params={ "cc": country, "id": app_id, "displayable-kind": 11, "startIndex": start, "endIndex": start + PAGE_SIZE, "sort": sort, "appVersion": "all", "bodyLengthLimit": 1000, }, headers=apple_headers(country), timeout=30, follow_redirects=True) try: return response.json()["userReviewList"] except (json.JSONDecodeError, KeyError): time.sleep(2 ** attempt * 2) return None def scrape_deep_reviews(app_id: str, country: str = "us", start: int = 0, limit: int = 300) -> list: seen, reviews = set(), [] for offset in range(start, start + limit, PAGE_SIZE): rows = fetch_review_rows(app_id, country, offset) if not rows: break for row in rows: if row["userReviewId"] in seen: continue seen.add(row["userReviewId"]) reviews.append({ "reviewId": row["userReviewId"], "userName": row["name"], "title": row["title"], "content": row["body"], "score": row["rating"], "voteSum": row["voteSum"], "voteCount": row["voteCount"], "updated": row["date"], }) return reviews reviews = scrape_deep_reviews("6448311069", "us", limit=300) print(len(reviews), reviews[0]) ``` In the example above, we pass `limit=300`, so `scrape_deep_reviews` walks three windows of 100 starting at indexes 0, 100 and 200. All three are sorted newest first with `sort=4`. Note that a window past the end of the list gets an empty `userReviewList` back, and that's where our loop stops. The `seen` set skips any review two windows share. Our script prints the review count and the first review: python ```python 300 {'reviewId': '14602501847', 'userName': 'Stirpad', 'title': 'Great', 'content': 'Chat gbt is tuff', 'score': 5, 'voteSum': 0, 'voteCount': 0, 'updated': '2026-09-28T07:50:46Z'} ``` Stirpad's review `14602501847` shows up in the RSS feed as well, this time with an `updated` of `2026-09-28T00:50:46-07:00`. Since the ID matches, merging the two sources is a de-duplication on `reviewId`. In September `-07:00` was the right offset. `00:50:46-07:00` and Stirpad's `07:50:46Z` are the same moment. On November 3, 2026, two days after daylight saving time ended, all 100 of the newest RSS reviews still ended in `-07:00`. Every one came out exactly an hour earlier than its `userReviewsRow` twin! When a review shows up in both, we keep its `Z` time. `userReviewsRow` also goes deeper than the RSS feed. On November 3, 2026, we asked it for a window starting at index 130,000 and still got 100 rows back, all from November 2024. The US total that day was 145,689. That total grows every day, so index 130,000 lands on newer reviews with each run. [How to Scrape Hidden APIsIn this tutorial we'll be taking a look at scraping hidden APIs which are becoming more and more common in modern dynamic websites - what's the best way to scrape them?](https://scrapfly.io/blog/posts/how-to-scrape-hidden-apis) Scrapfly #### Scale your web scraping effortlessly Scrapfly handles proxies, browsers, and anti-bot bypass — so you can focus on data. [Try Free →](https://scrapfly.io/register)## How Do You Scrape an apps.apple.com Page With Scrapfly? Our scraper only reaches for the app page when Lookup returns no record. To retrieve it, all we need is the same app ID and country, as the page lives at `https://apps.apple.com/{country}/app/id{app_id}`. python ```python def _extract_ios_version(operating_system: Optional[str]) -> Optional[str]: """pull "17.0" out of a string like "Requires iOS 17.0 or later" """ if not operating_system: return None match = re.search(r"iOS[\s\u00a0]*([\d.]+)", operating_system) return match.group(1) if match else operating_system def parse_app_metadata_from_page(response: ScrapeApiResponse, app_id: str, country: str) -> Dict: """fallback parser: read the ld+json block embedded in the apps.apple.com page. Used when the iTunes lookup API has no data for the app/country combo. Only a subset of fields (compared to parse_app_metadata) is available here. """ script = response.selector.css('script#software-application[type="application/ld+json"]::text').get() data = json.loads(script) if script else {} if data.get("@type") != "SoftwareApplication" or not data.get("name"): raise ValueError(f"No app metadata found on apps.apple.com page for {app_id}") offers = data.get("offers") or {} rating = data.get("aggregateRating") or {} author = data.get("author") or {} price = offers.get("price") or 0 genres = [genre for genre in (data.get("applicationCategory"), data.get("applicationSubCategory")) if genre] devices = [device.strip() for device in (data.get("availableOnDevice") or "").split(",") if device.strip()] return { "appId": str(app_id), "bundleId": None, "title": data.get("name"), "description": data.get("description"), "summary": None, "developer": author.get("name"), "developerId": None, "developerUrl": author.get("url"), "seller": author.get("name"), "sellerUrl": None, "genre": genres[0] if genres else None, "genres": genres, "genreIds": [], "price": price, "free": price == 0, "currency": offers.get("priceCurrency"), "formattedPrice": "Free" if price == 0 else None, "score": rating.get("ratingValue"), "scoreCurrentVersion": rating.get("ratingValue"), "ratings": rating.get("reviewCount"), "ratingsCurrentVersion": rating.get("reviewCount"), "version": None, "released": None, "updated": None, "contentRating": None, "minimumOsVersion": _extract_ios_version(data.get("operatingSystem")), "fileSizeBytes": None, "languages": [], "icon": data.get("image"), "screenshots": [], "ipadScreenshots": [], "appletvScreenshots": [], "supportedDevices": devices, "advisories": [], "features": [], "url": response.scrape_result.get("url") or response.config["url"], "country": country.lower(), } ``` On ChatGPT's page, our fallback parser produced these fields, among others: python ```python {'title': 'ChatGPT', 'developer': 'OpenAI OpCo, LLC', 'score': 4.8, 'ratings': 10778636, 'minimumOsVersion': '18.0', 'genres': ['BusinessApplication'], 'supportedDevices': ['iPhone', 'iPad']} ``` However, not all of these values match Lookup's - the page rounds the rating to `4.8`, and its genre is the schema.org category `BusinessApplication` rather than Lookup's `Productivity`. That being said, `ratings` matched the Lookup count in this run, though on November 5, 2026 the page was 32,137 ratings ahead of Lookup. We can see that now our scraper has three sources feeding records in, though two of them use different field names for the same review. ## How Do You Normalize Apple App Store Metadata and Customer Reviews? Since three sources now feed our scraper, we'll keep its output down to two record types - app metadata and reviews - joined by `appId` and `country`. Most of that mapping is already done, as `parse_app_metadata` and `parse_app_metadata_from_page` return the same 37 keys, while `scrape_deep_reviews` maps its rows onto the RSS review keys. Here's how the source fields line up: | Source field | Normalized key | Type | Nullable | |---|---|---|---| | Lookup `trackId` | `appId` | string | no | | Lookup `averageUserRating` | `score` | float | yes | | Lookup `userRatingCount` | `ratings` | int | yes | | Lookup `screenshotUrls` | `screenshots` | list | empty for some apps | | RSS `id.label` / rows `userReviewId` | `reviewId` | string | no | | RSS `im:rating.label` / rows `rating` | `score` | int | no | | RSS `updated.label` / rows `date` | `updated` | ISO 8601 string | no | | RSS `im:version.label` | `version` | string | missing from rows | So, we'll have to convert every `updated` value in the table above to UTC before merging the two sources. Neither review endpoint returns the country or a scrape timestamp, and our review records are missing the app ID too. So, we'll add all three to every review ourselves. With that, both record types can even share a single JSON Lines file! [JSONL vs JSONLearn the differences between JSON and JSONLines, their use cases, and efficiency. Why JSONLines excels in web scraping and real-time processing](https://scrapfly.io/blog/posts/jsonl-vs-json) ## How Do You Handle Empty Apple App Store Results and Endpoint Changes? When scraping Apple's endpoints, we can't count on an error status whenever a response comes back empty. Apart from the two HTTP 400 cases we've seen, on RSS page 11 and the `143444-1` UK storefront, Apple answered every request so far with HTTP 200. That includes the responses with no data in them at all. In testing we ran into four of these empty responses, and each one needs a different reaction from our scraper. So, let's write a `classify` function that reads the body for each surface instead: python ```python def classify(surface: str, body: str) -> tuple[str, str]: try: payload = json.loads(body) except json.JSONDecodeError: return "error", "store error page, check the X-Apple-Store-Front header" if surface == "lookup": return ("success", "app found") if payload.get("resultCount") else ("empty", "resultCount 0") if surface == "rss": entries = payload.get("feed", {}).get("entry") return ("success", "feed has entries") if entries else ("retry", "empty feed page") if surface == "rows": rows = payload.get("userReviewList") return ("success", f"{len(rows)} rows") if rows else ("empty", "end of review list") return "success", "payload parsed" ``` Let's hand `classify` a good and a bad response from both Lookup and `userReviewsRow` and see how it sorts them: python ```python LOOKUP_URL = "https://itunes.apple.com/lookup" def get_rows(start: int, headers: dict) -> httpx.Response: params = {"cc": "us", "id": "6448311069", "displayable-kind": 11, "startIndex": start, "endIndex": start + PAGE_SIZE, "sort": 4, "appVersion": "all", "bodyLengthLimit": 1000} return httpx.get(ROWS_URL, params=params, headers=headers, timeout=30, follow_redirects=True) cases = { "lookup ok": ("lookup", httpx.get(LOOKUP_URL, params={"id": "6448311069", "country": "us"})), "lookup unknown": ("lookup", httpx.get(LOOKUP_URL, params={"id": "1", "country": "us"})), "rows ok": ("rows", get_rows(0, apple_headers("us"))), "rows no header": ("rows", get_rows(0, {})), "rows past end": ("rows", get_rows(200000, apple_headers("us"))), } for name, (surface, response) in cases.items(): print(f"{name:<15} {response.status_code} {len(response.text):>7}", *classify(surface, response.text), sep=" ") ``` Which gives us: text ```text lookup ok 200 10201 success app found lookup unknown 200 42 empty resultCount 0 rows ok 200 111483 success 100 rows rows no header 200 4980 error store error page, check the X-Apple-Store-Front header rows past end 200 21 empty end of review list ``` The `rows no header` row is the 4,980-character store error page we got without the `X-Apple-Store-Front` header. Retrying it won't help, as Apple keeps sending that same page until we fix the header. [Rate limiting](https://scrapfly.io/blog/posts/what-is-http-error-429-too-many-requests) is something we'd normally spot by its status code as well. Apple sent HTTP 200 for every case in our output though, so the response body is all our scraper has to go on! ## Scraping the App Store With Scrapfly Scrapfly's [Web Scraping API](https://scrapfly.io/products/web-scraping-api) is a single HTTP endpoint for collecting web data at scale, with a **98% success rate** on the hardest public pages and proxies across **190+ countries**. - [Web Unblocker](https://scrapfly.io/products/unblocker) (previously ASP) - handles 25+ anti-bot systems, including Cloudflare, DataDome, PerimeterX and Akamai. - [Smart proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) - residential and datacenter pools with country and ASN level geo-targeting. - [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - render SPAs and dynamic pages through real cloud browsers. - [Browser automation scenarios](https://scrapfly.io/docs/scrape-api/javascript-scenario) - scroll, click, fill forms, and wait for elements without managing a browser fleet. - [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - return pages as HTML, JSON, clean text, or LLM ready Markdown. - [Session management](https://scrapfly.io/docs/scrape-api/session) - keep cookies, headers, and IPs consistent across multi step flows. - [Smart caching](https://scrapfly.io/docs/scrape-api/getting-started#api_param_cache) - cache successful responses to cut cost on repeat scraping jobs. - [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Rust](https://scrapfly.io/docs/sdk/rust), and [no-code integrations](https://scrapfly.io/docs/integration/getting-started) including [Make](https://scrapfly.io/integration/make), [n8n](https://scrapfly.io/integration/n8n), [Zapier](https://scrapfly.io/integration/zapier), [LangChain](https://scrapfly.io/integration/langchain), and [LlamaIndex](https://scrapfly.io/integration/llamaindex). ### Web Scraping API Scrape any website with our powerful API. Anti-bot bypass, JavaScript rendering, and rotating proxies built-in. [Try Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started) ## FAQ Is the iTunes Lookup API the same as the App Store Connect API?No. The Lookup API we used here is public and returns storefront metadata for any app ID without any authentication. [App Store Connect](https://developer.apple.com/documentation/appstoreconnectapi/customer-reviews) on the other hand is for apps your team manages, and all of its resources are behind authentication. Do you need JavaScript rendering to scrape apps.apple.com?No. The JSON-LD block is already in the app page's initial HTML, so our `parse_app_metadata_from_page` parser can read it without `render_js`. As for metadata and reviews, they come from JSON endpoints, which don't need a browser either. What is the Google Play counterpart to this Apple App Store scraper?Google Play is the Android counterpart, and the card near the top of this article points to our guide for scraping it. Note that Google Play uses different app identifiers, endpoints, pagination and review schemas than the App Store. Is scraping Apple App Store reviews legal?Collecting public app listings and reviews is generally allowed, though Apple's terms and local rules still apply. That being said, we recommend checking both before running a large job, and not republishing review text as your own. ## Summary We've built an Apple App Store scraper which collects app metadata as well as customer reviews from several sources rather than starting from the app page. For this, we've used the iTunes Lookup API for metadata, the RSS feed for the newest 500 reviews and the `userReviewsRow` endpoint for the rest. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents - [Key Takeaways](#key-takeaways) - [Project Setup](#project-setup) - [What Apple App Store Data Can You Scrape?](#what-apple-app-store-data-can-you-scrape) - [How Do You Find an Apple App Store App ID and Country Storefront?](#how-do-you-find-an-apple-app-store-app-id-and-country-storefront) - [How Do You Scrape App Metadata With the iTunes Lookup API?](#how-do-you-scrape-app-metadata-with-the-itunes-lookup-api) - [How Do You Scrape Apple App Store Customer Reviews?](#how-do-you-scrape-apple-app-store-customer-reviews) - [How do you parse customer reviews from the RSS/JSON feed?](#how-do-you-parse-customer-reviews-from-the-rss-json-feed) - [How should you paginate and de-duplicate Apple App Store reviews?](#how-should-you-paginate-and-de-duplicate-apple-app-store-reviews) - [How do you collect more than 500 Apple App Store reviews?](#how-do-you-collect-more-than-500-apple-app-store-reviews) - [How Do You Scrape an apps.apple.com Page With Scrapfly?](#how-do-you-scrape-an-apps-apple-com-page-with-scrapfly) - [How Do You Normalize Apple App Store Metadata and Customer Reviews?](#how-do-you-normalize-apple-app-store-metadata-and-customer-reviews) - [How Do You Handle Empty Apple App Store Results and Endpoint Changes?](#how-do-you-handle-empty-apple-app-store-results-and-endpoint-changes) - [Scraping the App Store With Scrapfly](#scraping-the-app-store-with-scrapfly) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ python hidden-api ### How to Scrape Google Play App Reviews and Data Scrape Google Play app metadata, ratings, and the full review set with Python, past the few-hundred-review ceiling the f... ](https://scrapfly.io/blog/posts/how-to-scrape-google-play-app-reviews-and-data) [ python data-parsing ### Ultimate Guide to JSON Parsing in Python Learn JSON parsing in Python with this ultimate guide. Explore basic and advanced techniques using json, and tools like ... ](https://scrapfly.io/blog/posts/how-to-use-python-to-parse-json) [ python hidden-api ### How to Scrape YouTube in 2026 Learn how to scrape YouTube channel, video, comment, and Shorts data in Python using hidden APIs and yt-dlp. No API key ... ](https://scrapfly.io/blog/posts/how-to-scrape-youtube) Scale your web scraping effortlessly, **1,000 free credits** [Start Free](https://scrapfly.io/register)