[Blog](https://scrapfly.io/blog) / [python](https://scrapfly.io/blog/tag/python) / [How to Scrape Reddit Posts, Subreddits and Profiles](https://scrapfly.io/blog/posts/how-to-scrape-reddit-social-data) # How to Scrape Reddit Posts, Subreddits and Profiles by [Mazen Ramadan](https://scrapfly.io/blog/author/mazen) Oct 22, 2026 26 min read [\#python](https://scrapfly.io/blog/tag/python) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-reddit-social-data "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-reddit-social-data&text=How%20to%20Scrape%20Reddit%20Posts%2C%20Subreddits%20and%20Profiles "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-reddit-social-data "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-reddit-social-data) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-reddit-social-data) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-reddit-social-data) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-reddit-social-data) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-reddit-social-data) A plain HTTP request to a Reddit post page returned HTTP 200 and 8,483 bytes of HTML: a page titled "Reddit" with a hidden form carrying a `js_challenge` field, and one script that computes a value, writes it into the form's `solution` field and submits it. It contained zero `shreddit-post` elements. The same URL fetched through the `scrapfly-scrapers` Reddit scraper returned the rendered post, and the scraper's four test jobs (subreddits, posts with comments, profile posts, profile comments) all passed in our run on October 20, 2026. This guide walks through that maintained scraper, function by function, and shows the output it produced. ## Key Takeaways - A direct request with a Chrome User-Agent got a 200 response carrying a JavaScript challenge form instead of content, so a 200 status alone does not prove you got Reddit content. - The canonical scraper sends every request through Scrapfly with four settings: `unblocker=True`, `render_js=True`, `country="US"` and `proxy_pool="public_residential_pool"`, and it targets `www.reddit.com` for every page type. - Post comments come from Reddit's `/svc/shreddit/comments/` endpoint, fetched once per sort order and merged by comment ID, so you get a sample of a thread rather than every comment. - On October 20, 2026 the subreddit job returned 75 posts, the post job 57 top-level comments (74 with replies), the profile job 78 posts and the profile comments job 3 comments, and each passed its schema and minimum-count checks. - Validate every run against a schema and a minimum item count. Selectors, cursors and nullable fields are where Reddit scrapers break silently. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. [**Reddit Scraper**github.com/scrapfly/scrapfly-scrapers/tree/main/reddit-scraper](https://github.com/scrapfly/scrapfly-scrapers/tree/main/reddit-scraper) ## Does Reddit Scraping Still Work in 2026? Yes, if the request route can get past the challenge page. From our test host, plain requests to subreddit, post and profile URLs all came back as the same 200 challenge page with no Reddit content in it. The Scrapfly route in the canonical scraper returned rendered pages for all three. If Reddit's official Data API and its Python wrapper [PRAW](https://praw.readthedocs.io/en/stable/) already cover your use case and rate limits, keep using them. This guide covers the other case: public subreddit listings, post threads and profile activity collected from the pages a logged-out browser sees. ### Which Reddit Scraping Method Does This Guide Use? Every code block below is copied from the [scrapfly-scrapers Reddit scraper](https://github.com/scrapfly/scrapfly-scrapers/tree/main/reddit-scraper) at a pinned commit, so there is one implementation to keep in sync. A plain [How to Web Scrape with HTTPX and Python](https://scrapfly.io/blog/posts/web-scraping-with-python-httpx) request shows up only as a diagnostic for what a direct request returns. Reddit announced it will shut down unauthenticated `.json` endpoints ([announcement](https://www.reddit.com/r/modnews/comments/1tq9vxo/protecting_communities_from_scrapers_and_platform/)). The scraper never uses them, so nothing below depends on that change. ### Which Reddit Data Can You Collect From Public Pages? The scraper returns four output families, all from pages a logged-out visitor can open: | In scope | Out of scope | |---|---| | Subreddit info and post listings | Private messages | | Post metadata and comment threads | Private or restricted communities | | A user's submitted posts | Login-only data | | A user's comments | Deleted content reconstruction | | | Exhaustive historical archives | Posts and comments are the usual input for text work such as [sentiment analysis](https://scrapfly.io/blog/posts/intro-to-using-web-scraping-for-sentiment-analysis). ## How Do You Set Up the Canonical Python Reddit Scraper? The scraper lives in the `reddit-scraper` directory of the repository. This guide matches commit `950fa25` (October 20, 2026). It declares Python `^3.10` and `scrapfly-sdk` `^0.12.0`. Our run used `scrapfly-sdk` 0.12.0 and `parsel` 1.12.1 on Python 3.10. Set your Scrapfly API key, then clone and install: shell ```shell export SCRAPFLY_KEY="YOUR SCRAPFLY KEY" git clone https://github.com/scrapfly/scrapfly-scrapers.git cd scrapfly-scrapers/reddit-scraper poetry install poetry run python run.py ``` `run.py` scrapes one example of each page type and writes the JSON to `./results`. ### Install Python and scrapfly-sdk for the Reddit Scraper Every request goes through one client and one shared config: python ```python import os import re from typing import Callable, Dict, List, Literal, Optional from loguru import logger as log from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse SCRAPFLY = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"]) BASE_CONFIG = { # enable the anti scraping protection "unblocker": True, # set the proxy country to US "country": "US", # bypassing reddit requires enabling JavaScript and using the residential proxy pool "render_js": True, "proxy_pool": "public_residential_pool", } REDDIT_URL = "https://www.reddit.com" ``` `unblocker` turns on the [Unblocker (previously ASP)](https://scrapfly.io/products/unblocker), which handles the challenge page. `render_js` runs the page in a browser so Reddit's web components exist in the HTML. `country` and `proxy_pool` send the request from US residential IPs. Per the maintainers' code comment, Reddit needs both JavaScript rendering and the residential pool. Reddit puts most values in attributes of custom elements and uses relative links, so the module starts with a few small helpers that the parsers below reuse: python ```python def absolute_url(path: Optional[str]) -> Optional[str]: """turn a reddit path into an absolute URL""" if not path: return None if path.startswith("http"): return path return REDDIT_URL + path if path.startswith("/") else f"{REDDIT_URL}/{path}" def to_int(value) -> Optional[int]: """parse an integer attribute, or None when missing""" return int(value) if value else None def parse_score(node) -> Optional[int]: """parse score from @score or the rendered vote button faceplate-number""" score = ( node.xpath("./@score").get() or node.xpath(".//shreddit-comment-action-row/@score").get() or node.xpath(".//span[contains(@class, 'rpl-vote-button-group')]//faceplate-number/@number").get() or node.xpath(".//button[@upvote]/following-sibling::span[1]//faceplate-number/@number").get() ) return to_int(score) def profile_url(author: Optional[str]) -> Optional[str]: """build a reddit user profile URL""" return f"{REDDIT_URL}/user/{author}" if author else None def comment_text(node) -> Optional[str]: """join paragraph text from a shreddit comment body""" parts = node.xpath(".//div[contains(@id, 'rtjson-content')]//p//text()").getall() text = "".join(parts).strip() return text or None def parse_attachment(post) -> tuple: """parse post attachment type and the best available media/content URL""" attachment_type = post.xpath("./@post-type").get() attachment_link = ( post.xpath(".//img[contains(@class, 'media-lightbox-img')]/@src").get() or post.xpath(".//shreddit-player/@preview").get() or post.xpath("./@content-href").get() ) return attachment_type, absolute_url(attachment_link) ``` ### Run the scrapfly-scrapers Reddit Tests Before Using the Output `test.py` has one test per output family. Each test scrapes live Reddit, validates every item against a Cerberus schema and asserts a minimum item count: | Test | Input | Minimum | Our run, Oct 20, 2026 | |---|---|---|---| | `test_subreddit_scraping` | r/wallstreetbets, 3 extra pages | 50 posts | 75 posts, passed | | `test_post_scraping` | one r/wallstreetbets post, sort `top` | 50 comments | 57 top-level, passed | | `test_user_post_scraping` | u/Scrapfly, sort `top`, 3 extra pages | 25 posts | 78 posts, passed | | `test_user_comment_scraping` | u/Scrapfly, sort `top`, 3 extra pages | 2 comments | 3 comments, passed | Install the dev dependencies and run all four, or one family at a time: shell ```shell poetry install --with dev poetry run pytest test.py poetry run pytest test.py -k test_subreddit_scraping poetry run pytest test.py -k test_post_scraping poetry run pytest test.py -k test_user_post_scraping poetry run pytest test.py -k test_user_comment_scraping ``` A pass shows the selectors and schema held on the day of the run. Run the tests again before you rely on output from a later day. ## How Do You Scrape Reddit Subreddits and Paginate Posts? A subreddit scrape returns two things: an `info` object for the community and a list of posts. `parse_subreddit()` builds both from one page of HTML and also returns the cursor for the next page. ### Parse shreddit-post Fields From a Reddit Subreddit Each post card is an `article[@data-post-id]` container with a `shreddit-post` custom element inside. The values you want sit on that element as attributes, not as visible text: `author`, `author-id`, `post-title`, `created-timestamp`, `score`, `comment-count` and `post-type`. The member count comes from the `faceplate-number` element followed by the members label. The header's `weekly-active-users` attribute is the fallback. python ```python def parse_subreddit(response: ScrapeApiResponse) -> Dict: """parse subreddit info and post cards from HTML""" selector = response.selector url = response.context["url"] members = selector.xpath("//faceplate-number[following-sibling::text()[contains(., 'members')]]/@number").get() weekly_active = selector.xpath("//shreddit-subreddit-header/@weekly-active-users").get() rank = selector.xpath("//strong[@id='position']/text()").get() bookmarks = {} for item in selector.xpath("//div[faceplate-tracker[@source='community_menu']]/faceplate-tracker"): name = item.xpath(".//a/span/span/span/text()").get() link = item.xpath(".//a/@href").get() if name and link: bookmarks[name] = link posts = [] for article in selector.xpath("//article[@data-post-id]"): post = article.xpath(".//shreddit-post") if not post: continue post = post[0] author = post.xpath("./@author").get() label = article.xpath(".//span[contains(@class, 'bg-tone-4')]/div/text()").get() attachment_type, attachment_link = parse_attachment(post) posts.append( { "authorProfile": profile_url(author), "authorId": post.xpath("./@author-id").get(), "title": post.xpath("./@post-title").get(), "link": absolute_url(article.xpath(".//a/@href").get()), "publishingDate": post.xpath("./@created-timestamp").get(), "postId": post.xpath("./@id").get(), "postLabel": label.strip() if label else None, "postUpvotes": to_int(post.xpath("./@score").get()), "commentCount": to_int(post.xpath("./@comment-count").get()), "attachmentType": attachment_type, "attachmentLink": attachment_link, } ) return { "info": { "id": url.split("/r")[-1].replace("/", ""), "description": selector.xpath("//shreddit-subreddit-header/@description").get(), "rank": rank.strip() if rank else None, "members": to_int(members) if members else to_int(weekly_active), "bookmarks": bookmarks, "url": url, }, "post_data": posts, "cursor": selector.xpath("//shreddit-post/@more-posts-cursor").get(), } ``` `rank`, `postLabel`, `attachmentLink` and the post's `authorId` are nullable in the test schema. In our run `rank` was `null` and `postLabel` was `null` on all 75 posts, so treat both as optional. The `bookmarks` dict holds whatever links the subreddit puts in its community menu, which differs per subreddit. ### Paginate Reddit Posts With more-posts-cursor The first page carries a `more-posts-cursor` attribute on its `shreddit-post` elements. `scrape_subreddit()` passes that cursor to Reddit's `community-more-posts` endpoint, which returns the next batch as HTML that `parse_subreddit()` can read. Calling a background endpoint like this instead of scrolling a browser is the pattern covered in [How to Scrape Hidden APIsIn this tutorial we'll be taking a look at scraping hidden APIs which are becoming more and more common in modern dynamic websites - what's the best way to scrape them?](https://scrapfly.io/blog/posts/how-to-scrape-hidden-apis) python ```python async def scrape_subreddit(subreddit_id: str, max_pages: int = None) -> Dict: """scrape posts from a subreddit""" log.info("scraping subreddit r/{}", subreddit_id) url = f"{REDDIT_URL}/r/{subreddit_id}/" response = await SCRAPFLY.async_scrape( ScrapeConfig(url, **BASE_CONFIG, wait_for_selector="//article[@data-post-id]") ) data = parse_subreddit(response) subreddit_data = {"info": data["info"], "posts": data["post_data"]} cursor = data["cursor"] while cursor and (max_pages is None or max_pages > 0): page_url = ( f"{REDDIT_URL}/svc/shreddit/community-more-posts/hot/" f"?after={cursor}%3D%3D&t=DAY&name={subreddit_id}&feedLength=3" ) response = await SCRAPFLY.async_scrape(ScrapeConfig(page_url, **BASE_CONFIG)) data = parse_subreddit(response) cursor = data["cursor"] subreddit_data["posts"].extend(data["post_data"]) if max_pages is not None: max_pages -= 1 log.success(f"scraped {len(subreddit_data['posts'])} posts from the subreddit: r/{subreddit_id}") return subreddit_data ``` The `name` parameter uses the `subreddit_id` you pass in, so every page stays on the same subreddit. The cursor is sent with an encoded `==` suffix (`%3D%3D`). The loop stops when a page has no cursor or `max_pages` runs out. `max_pages` counts pages after the first one, so `max_pages=3` requests up to four pages. Example Outputjson ```json { "info": { "id": "wallstreetbets", "description": "Like 4chan found a Bloomberg Terminal.", "rank": null, "members": 2308358, "bookmarks": { "Wiki": "/r/wallstreetbets/wiki/index/", "YouTube": "https://www.youtube.com/@WSBverse?sub_confirmation=1", "Discord": "https://discord.gg/wsbverse", "Twitch": "https://twitch.tv/wsbverse", "WallStreetBets.com": "https://WallStreetBets.com" }, "url": "https://www.reddit.com/r/wallstreetbets/" }, "posts": [ { "authorProfile": "https://www.reddit.com/user/APerson2021", "authorId": "t2_ae3tfo4m", "title": "Why did Ask Jeeves shut down during peak LLM mania?", "link": "https://www.reddit.com/r/wallstreetbets/comments/1xarwa2/why_did_ask_jeeves_shut_down_during_peak_llm_mania/", "publishingDate": "2026-10-20T13:17:04.433000+0000", "postId": "t3_1xarwa2", "postLabel": null, "postUpvotes": 1437, "commentCount": 224, "attachmentType": "image", "attachmentLink": "https://preview.redd.it/why-did-ask-jeeves-shut-down-during-peak-llm-mania-v0-qyvi7xrehmwh1.jpeg?auto=webp&s=3803ddcce96d7e83d5741e0f32c1435a122cf231" }, { "authorProfile": "https://www.reddit.com/user/Xx_KiK_xX", "authorId": "t2_3ihv3bxb", "title": "NVDA", "link": "https://www.reddit.com/r/wallstreetbets/comments/1wy1qzo/nvda/", "publishingDate": "2026-10-05T06:49:51.627000+0000", "postId": "t3_1wy1qzo", "postLabel": null, "postUpvotes": 27017, "commentCount": 1003, "attachmentType": "video", "attachmentLink": "https://v.redd.it/g5odx9anilth1/CMAF_96.mp4" }, ... ] } ``` ## How Do You Scrape Reddit Posts and Nested Comments? A post scrape makes two kinds of requests. The post page gives you the post's metadata. The comments come from a separate endpoint, `/svc/shreddit/comments/r/{subreddit}/{post_id}`, which returns `shreddit-comment` elements as HTML. ### Parse Reddit Post Metadata From shreddit-post The post page has one `shreddit-post` element, and `parse_post_info()` reads its attributes: python ```python def parse_post_info(response: ScrapeApiResponse) -> Dict: """parse post data from a subreddit post page""" selector = response.selector author = selector.xpath("//shreddit-post/@author").get() label = selector.xpath("//faceplate-tracker[@source='post']/a/span/div/text()").get() subreddit = selector.xpath("//shreddit-post/@subreddit-prefixed-name").get() return { "authorId": selector.xpath("//shreddit-post/@author-id").get(), "author": author, "authorProfile": profile_url(author), "subreddit": subreddit.replace("r/", "") if subreddit else None, "postId": selector.xpath("//shreddit-post/@id").get(), "postLabel": label.strip() if label else None, "publishingDate": selector.xpath("//shreddit-post/@created-timestamp").get(), "postTitle": selector.xpath("//shreddit-post/@post-title").get(), "postLink": absolute_url(selector.xpath("//shreddit-post/@permalink").get()), "commentCount": to_int(selector.xpath("//shreddit-post/@comment-count").get()), "upvoteCount": to_int(selector.xpath("//shreddit-post/@score").get()), "attachmentType": selector.xpath("//shreddit-post/@post-type").get(), "attachmentLink": selector.xpath("//shreddit-post/@content-href").get(), } ``` `subreddit` is checked before `.replace()` runs, so a missing `subreddit-prefixed-name` attribute gives `None` instead of an exception. ### Parse Reddit Comments and Rebuild the Reply Tree Each `shreddit-comment` element has a `thingid` and a `parentid`. `parse_post_comments()` stores every comment by ID, then attaches each one to its parent's `replies` list. Comments whose parent is not in the response become top-level entries. Authors shown as `[deleted]` become `None`. python ```python def parse_post_comments(response: ScrapeApiResponse) -> List[Dict]: """parse post comments from shreddit-comment elements, rebuilding the reply tree""" comments = {} parent_ids = {} order = [] for node in response.selector.xpath("//shreddit-comment"): thing_id = node.xpath("./@thingid").get() if not thing_id or thing_id in comments: continue author = node.xpath("./@author").get() if author == "[deleted]": author = None comments[thing_id] = { "authorId": node.xpath("./@author-id").get() or node.xpath(".//shreddit-overflow-menu/@author-id").get(), "author": author, "authorProfile": profile_url(author), "commentId": thing_id, "link": absolute_url(node.xpath("./@permalink").get()), "publishingDate": node.xpath("./@created").get(), "commentBody": comment_text(node), "upvotes": parse_score(node), } parent_ids[thing_id] = node.xpath("./@parentid").get() order.append(thing_id) top_level = [] for thing_id in order: comment = comments[thing_id] parent_id = parent_ids[thing_id] if parent_id and parent_id in comments: comments[parent_id].setdefault("replies", []).append(comment) else: top_level.append(comment) return top_level ``` `scrape_post()` fetches the post page, then requests the comments endpoint for the sort you asked for plus `top`, `new` and `old`, skipping duplicates, so `sort="top"` makes three comments requests. It keeps the first copy of each comment ID: python ```python async def scrape_post_comments(subreddit: str, post_id: str, sort: str) -> List[Dict]: """fetch and parse one sort's worth of comments via the shreddit comments endpoint""" url = f"{REDDIT_URL}/svc/shreddit/comments/r/{subreddit}/{post_id}?sort={sort}" response = await SCRAPFLY.async_scrape(ScrapeConfig(url, **BASE_CONFIG)) return parse_post_comments(response) async def scrape_post(url: str, sort: Literal["old", "new", "top"]) -> Dict: """scrape subreddit post and comment data""" log.info("scraping post {}", url) response = await SCRAPFLY.async_scrape( ScrapeConfig(url, **BASE_CONFIG, wait_for_selector="//shreddit-post") ) post_data = {"info": parse_post_info(response)} comments_by_id = {} for comment_sort in dict.fromkeys([sort, "top", "new", "old"]): for comment in await scrape_post_comments( post_data["info"]["subreddit"], post_data["info"]["postId"], comment_sort ): comments_by_id.setdefault(comment["commentId"], comment) post_data["comments"] = list(comments_by_id.values()) log.success(f"scraped {len(post_data['comments'])} comments from the post {url}") return post_data ``` The scraper makes one comments request per sort and does not request further comment pages, so the result is a sample of the thread. On the test post, which reports 8,295 comments, our run collected 57 top-level comments and 74 including replies. Size downstream work to the comments you actually get, not to `commentCount`. Two output details to plan for. `authorId` was `null` on all 57 top-level comments in our run, so key comments by `author` or `commentId`. `comment_text()` joins every paragraph under a comment node, and replies are nested inside their parent's node, so a parent's `commentBody` can include its replies' text. Example Outputjson ```json { "info": { "authorId": "t2_qbvp0eq8b", "author": "wsbapp", "authorProfile": "https://www.reddit.com/user/wsbapp", "subreddit": "wallstreetbets", "postId": "t3_1c4vwlp", "postLabel": null, "publishingDate": "2024-04-15T20:00:20.757000+0000", "postTitle": "What Are Your Moves Tomorrow, April 16, 2024", "postLink": "https://www.reddit.com/r/wallstreetbets/comments/1c4vwlp/what_are_your_moves_tomorrow_april_16_2024/", "commentCount": 8295, "upvoteCount": 331, "attachmentType": "customPost", "attachmentLink": "https://www.reddit.com/r/wallstreetbets/comments/1c4vwlp/what_are_your_moves_tomorrow_april_16_2024/" }, "comments": [ { "authorId": null, "author": "Maverick2937474838", "authorProfile": "https://www.reddit.com/user/Maverick2937474838", "commentId": "t1_kzq97qi", "link": "https://www.reddit.com/r/wallstreetbets/comments/1c4vwlp/comment/kzq97qi/", "publishingDate": "2024-04-15T20:03:15.847000+0000", "commentBody": null, "upvotes": 164 }, { "authorId": null, "author": "billyd1984texas", "authorProfile": "https://www.reddit.com/user/billyd1984texas", "commentId": "t1_kzqi6y1", "link": "https://www.reddit.com/r/wallstreetbets/comments/1c4vwlp/comment/kzqi6y1/", "publishingDate": "2024-04-15T20:53:20.253000+0000", "commentBody": "Isreal needs to chill tf out. If they're so mad make a dis track like adults", "upvotes": 108 }, ... ] } ``` ## How Do You Scrape Reddit Profile Posts and Comments? Profiles have two routes on `www.reddit.com`: `/user/{username}/submitted/` for posts and `/user/{username}/comments/` for comments. Both paginate the same way. Each page includes a `faceplate-partial` element whose `src` contains `-more-posts`, and that URL returns the next batch as HTML. `scrape_paginated()` follows it: python ```python def next_profile_page_url(selector) -> Optional[str]: """extract the next profile pagination URL from a faceplate-partial cursor link""" partial = selector.xpath("//faceplate-partial[contains(@src, '-more-posts')]/@src").get() return absolute_url(partial) async def scrape_paginated( url: str, parse: Callable[[ScrapeApiResponse], list], wait_for_selector: str = None, max_pages: int = None, ) -> list: """scrape a first page then follow faceplate-partial pagination""" config = {**BASE_CONFIG} if wait_for_selector: config["wait_for_selector"] = wait_for_selector response = await SCRAPFLY.async_scrape(ScrapeConfig(url, **config)) data = parse(response) next_url = next_profile_page_url(response.selector) while next_url and (max_pages is None or max_pages > 0): response = await SCRAPFLY.async_scrape(ScrapeConfig(next_url, **BASE_CONFIG)) data.extend(parse(response)) next_url = next_profile_page_url(response.selector) if max_pages is not None: max_pages -= 1 return data ``` Profile data is tied to a named person. Collect only what your use case needs, and do not use it for outreach or re-identification. ### Paginate Reddit Profile Posts Profile posts reuse the `shreddit-post` element and read the same attributes as subreddit cards, under slightly different field names, plus the post's subreddit: python ```python def parse_user_posts(response: ScrapeApiResponse) -> List[Dict]: """parse user posts from a profile page or its pagination partial""" data = [] for post in response.selector.xpath("//shreddit-post"): author = post.xpath("./@author").get() attachment_type, attachment_link = parse_attachment(post) data.append( { "authorId": post.xpath("./@author-id").get(), "author": author, "authorProfile": profile_url(author), "postId": post.xpath("./@id").get(), "postLink": absolute_url(post.xpath("./@permalink").get()), "postTitle": post.xpath("./@post-title").get(), "postSubreddit": post.xpath("./@subreddit-prefixed-name").get(), "publishingDate": post.xpath("./@created-timestamp").get(), "commentCount": to_int(post.xpath("./@comment-count").get()), "postScore": to_int(post.xpath("./@score").get()), "attachmentType": attachment_type, "attachmentLink": attachment_link, } ) return data async def scrape_user_posts( username: str, sort: Literal["new", "top", "controversial"], max_pages: int = None ) -> List[Dict]: """scrape user posts""" log.info("scraping posts from user {}", username) url = f"{REDDIT_URL}/user/{username}/submitted/?sort={sort}" post_data = await scrape_paginated( url, parse_user_posts, wait_for_selector="//shreddit-post", max_pages=max_pages ) log.success(f"scraped {len(post_data)} posts from the {username} reddit profile") return post_data ``` Each `postLink` is a post URL you can pass to `scrape_post()` for its comments. For crawling links like that at scale, see [How to Crawl the Web with PythonIntroduction to web crawling with Python. What is web crawling? How it differs from web scraping? And a deep dive into code, building our own crawler and an example project crawling Shopify-powered websites.](https://scrapfly.io/blog/posts/crawling-with-python) Example Outputjson ```json [ { "authorId": "t2_saicgkn9", "author": "Scrapfly", "authorProfile": "https://www.reddit.com/user/Scrapfly", "postId": "t3_1h7ajw1", "postLink": "https://www.reddit.com/r/scrapfly/comments/1h7ajw1/comprehensive_guide_to_okhttp_for_java_and_kotlin/", "postTitle": "Comprehensive Guide to OkHttp for Java and Kotlin", "postSubreddit": "r/scrapfly", "publishingDate": "2024-12-05T14:35:44.960000+0000", "commentCount": 0, "postScore": 3, "attachmentType": "link", "attachmentLink": "https://scrapfly.io/blog/guide-to-okhttp-java-kotlin/" }, { "authorId": "t2_saicgkn9", "author": "Scrapfly", "authorProfile": "https://www.reddit.com/user/Scrapfly", "postId": "t3_1fv3am2", "postLink": "https://www.reddit.com/r/scrapfly/comments/1fv3am2/how_to_use_curl_get_requests/", "postTitle": "How to Use cURL GET Requests", "postSubreddit": "r/scrapfly", "publishingDate": "2024-10-03T08:54:40.488000+0000", "commentCount": 0, "postScore": 3, "attachmentType": "link", "attachmentLink": "https://scrapfly.io/blog/how-to-use-curl-get-requests/" }, ... ] ``` ### Parse Reddit Profile Comments Profile comments render as `shreddit-profile-comment` elements. Besides the comment, each one links to the post it replied to, which ends up in `replyTo`: python ```python def parse_user_comments(response: ScrapeApiResponse, username: str) -> List[Dict]: """parse user comments from a profile page or its pagination partial""" data = [] for box in response.selector.xpath("//shreddit-profile-comment"): href = box.xpath("./@href").get() author = box.xpath(".//shreddit-overflow-menu/@author-name").get() or username post_title = "".join(box.xpath(".//h2//a//text()").getall()).strip() post_link = box.xpath(".//h2//a/@href").get() subreddit_match = re.match(r"^(/r/[^/]+)/", href or "") data.append( { "authorId": box.xpath(".//shreddit-overflow-menu/@author-id").get(), "author": author, "authorProfile": profile_url(author), "commentId": box.xpath("./@comment-id").get(), "commentLink": absolute_url(href), "commentBody": comment_text(box), "attachedCommentLinks": [ link for link in box.xpath(".//div[contains(@id, 'rtjson-content')]//a/@href").getall() if link and "reddit.com" not in link ], "publishingDate": box.xpath(".//faceplate-timeago/@ts").get(), "upvotes": parse_score(box), "replyTo": { "postTitle": post_title or None, "postLink": absolute_url(post_link), "postSubreddit": subreddit_match.group(1).lstrip("/") if subreddit_match else None, }, } ) return data async def scrape_user_comments( username: str, sort: Literal["new", "top", "controversial"], max_pages: int = None ) -> List[Dict]: """scrape user comments""" log.info("scraping comments from user {}", username) url = f"{REDDIT_URL}/user/{username}/comments/?sort={sort}" comment_data = await scrape_paginated( url, lambda response: parse_user_comments(response, username), wait_for_selector="//shreddit-profile-comment", max_pages=max_pages, ) log.success(f"scraped {len(comment_data)} comments from the {username} reddit profile") return comment_data ``` `attachedCommentLinks` keeps links in the comment body that point outside Reddit. `upvotes` is the single score Reddit shows for the comment, not separate upvote and downvote totals. Example Outputjson ```json [ { "authorId": "t2_saicgkn9", "author": "Scrapfly", "authorProfile": "https://www.reddit.com/user/Scrapfly", "commentId": "t1_kzry9ar", "commentLink": "https://www.reddit.com/r/webscraping/comments/1c4jd72/comment/kzry9ar/?context=3", "commentBody": "You can check out our web scraping academy resource, https://scrapfly.io/academy (it's totally free and independent of our service), which has a visual roadmap and allows you to learn/dig branch by branch. I hope this helps!\n \n Also, the website in the resources link https://webscraping.fyi/ looks very neat", "attachedCommentLinks": [ "https://scrapfly.io/academy", "https://webscraping.fyi/" ], "publishingDate": "2024-04-16T02:47:18.990000+0000", "upvotes": 8, "replyTo": { "postTitle": "Where to begin Web Scraping", "postLink": "https://www.reddit.com/r/webscraping/comments/1c4jd72/where_to_begin_web_scraping/", "postSubreddit": "r/webscraping" } }, ... ] ``` ## Why Does Reddit Return 403 or Empty Scraper Output? Check these five failure patterns first, and identify which one you have before you change code: - **Block page instead of content.** A 403, or a 200 whose body is a short challenge page with no `shreddit-post` elements. - **Rendered page, missing parser nodes.** HTTP 200 and real Reddit HTML, but zero `article[@data-post-id]` or `shreddit-post` matches. Check that the page you asked for has the content before changing selectors. - **Missing or malformed pagination cursor.** The first page parses and no second batch arrives. Log the cursor and the generated URL together. - **The unblocker could not solve a request.** The SDK raises `ScrapflyAspError` with `ERR::ASP::SHIELD_PROTECTION_FAILED` and HTTP 422. The error text says to retry after a few seconds, and the repository's tests retry each failed job three times, 30 seconds apart. - **Deleted or hidden fields.** Author, body, score or link comes back `null`. Keep the `null` instead of filling in a default. On r/ClaudeCode, a developer asked how to pull posts and comments from niche communities, structured by title, upvotes and theme, before passing them to an LLM ([thread](https://www.reddit.com/r/ClaudeCode/comments/1sjl56y/best_way_to_scrape_reddit_posts/)). A job like that can return an empty or partial list on a 200 response, which is why the checks below look at content and schema, not the status code. ### HTTP 403 or a 200 Challenge Page Check four things before you parse: status code, content type, response size and a selector you know should exist. Our direct request to a post page returned `200`, `text/html` and 8,483 bytes with no `shreddit-post` element. Through the scraper, the same post URL returned the rendered `shreddit-post` element that `parse_post_info()` reads. Call it bot protection unless you have captured a vendor's signature on the exact hostname. A plain request gets HTTP 200 and this challenge page, not the posts### Missing shreddit-post Selectors or Pagination Cursors Treat an empty list as a failure, not as an empty subreddit. The tests do this with minimum counts: at least 50 subreddit posts across four pages, 50 post comments, 25 profile posts and 2 profile comments. In your own runs, assert that `article[@data-post-id]` matched before you trust a subreddit page, and log the cursor next to the page URL so a broken pagination request is visible. For writing and testing selectors, see [Parsing HTML with XpathIntroduction to xpath in the context of web-scraping. How to extract data from HTML documents using xpath, best practices and available tools.](https://scrapfly.io/blog/posts/parsing-html-with-xpath) ### Deleted Reddit Authors and Nullable Fields Deleted and removed content is routine on Reddit. In our post run, 12 of 57 top-level comments had no author and 6 had no body. The schemas in `test.py` mark these fields nullable: comment `author`, `authorId`, `authorProfile`, `commentBody` and `upvotes`, and post `postLabel`, `postLink` and `attachmentLink`. Write consumers that accept `None` in those fields, and keep any schema you maintain in step with `test.py`. Scrapfly #### Scale your web scraping effortlessly Scrapfly handles proxies, browsers, and anti-bot bypass — so you can focus on data. [Try Free →](https://scrapfly.io/register)## How Do You Keep a Python Reddit Scraper Working? Treat the scraper as code with a test contract: - Keep endpoints and selectors in one module, as `reddit.py` does. - Store sanitized fixtures only where you are allowed to. - Run the per-page-type tests on a schedule that fits how often you scrape. - Alert on schema and item-count failures, not on HTTP status alone. ### Pin the scrapfly-scrapers Commit and Dependency Versions The code in this article matches one state of the repository: - Commit `950fa25` (October 20, 2026). The Reddit scraper last changed in `03f3bd7` on October 19, 2026, when it moved to `unblocker=True` and `scrapfly-sdk` `^0.12.0`. - Python `^3.10`, `scrapfly-sdk` `^0.12.0` (0.12.0 in our run), `parsel` `^1.8.1` - Run on October 20, 2026 against `www.reddit.com` subreddit, post, comments-endpoint and profile routes - `unblocker=True`, `render_js=True`, `country="US"`, `proxy_pool="public_residential_pool"` When you pull a newer commit, run the tests before you trust its output. ### Validate Reddit Output Schemas in CI The repository runs each test function as a separate nightly CI job, so a profile failure does not hide a passing subreddit job. Treat one failed run as a reason to look, not as proof that every route broke: read the failing assertion or error, then compare it with the runs before and after. These are the helpers `test.py` uses, followed by the profile comments test: python ```python from cerberus import Validator as _Validator import pytest import reddit import pprint pp = pprint.PrettyPrinter(indent=4) class Validator(_Validator): def _validate_min_presence(self, min_presence, field, value): pass # required for adding non-standard keys to schema def validate_or_fail(item, validator): if not validator.validate(item): pp.pformat(item) pytest.fail(f"Validation failed for item: {pp.pformat(item)}\nErrors: {validator.errors}") def require_min_presence(items, key, min_perc=0.1): """check whether dataset contains items with some amount of non-null values for a given key""" count = sum(1 for item in items if key in item and item[key] is not None) if count < len(items) * min_perc: pytest.fail( f'inadequate presence of "{key}" field in dataset, only {count} out of {len(items)} items have it (expected {min_perc*100}%)' ) ``` python ```python @pytest.mark.asyncio @pytest.mark.flaky(reruns=3, reruns_delay=30) async def test_user_comment_scraping(): user_comment_data = await reddit.scrape_user_comments( username="Scrapfly", sort="top", max_pages=3 ) validator = Validator(user_comment_schema, allow_unknown=True) for item in user_comment_data: validate_or_fail(item, validator) for k in user_comment_schema: require_min_presence(user_comment_data, k, min_perc=user_comment_schema[k].get("min_presence", 0.05)) assert len(user_comment_data) >= 2 ``` `allow_unknown=True` lets the scraper add fields without failing the test. `require_min_presence()` fails the run when too few items have a value for a key, which catches a selector that quietly stopped matching. ## FAQ Can You Scrape Reddit Without an API Key?Public Reddit pages don't need a Reddit API key, and the canonical scraper never calls Reddit's API. Access still depends on the route. From our test host, a direct request returned a 200 challenge page with no Reddit content, while the same URL through Scrapfly returned the rendered page. The Scrapfly route needs a Scrapfly API key in `SCRAPFLY_KEY`. Why Does Reddit Block httpx or Requests?A plain request with a Chrome User-Agent got HTTP 200 and an 8 KB page whose script computes a value and submits a hidden form. An HTTP client without a JavaScript engine never gets past that page, so you see the challenge instead of `shreddit-post` elements. We did not identify which signals Reddit checks, so don't assume a User-Agent change is enough. ## Summary: Choose a Reddit Access Route and Test It Pick the route first: the official API through PRAW if it covers what you need, or the canonical scraper for public pages. Then run the matching test, validate the schema and trust only the output that test produced. The canonical repository sends every request through the Scrapfly [Web Scraping API](https://scrapfly.io/products/web-scraping-api) because direct requests got the challenge page. The full source is in the [scrapfly-scrapers reddit-scraper](https://github.com/scrapfly/scrapfly-scrapers/tree/main/reddit-scraper) directory. ### Web Scraping API Scrape any website with our powerful API. Anti-bot bypass, JavaScript rendering, and rotating proxies built-in. [Try Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started) Similar guides exist for [How to Scrape Twitter (X.com) with Python in 2026](https://scrapfly.io/blog/posts/how-to-scrape-twitter), [How to Scrape Instagram in 2026](https://scrapfly.io/blog/posts/how-to-scrape-instagram), and [How to scrape Threads by Meta using Python (2026 Update)](https://scrapfly.io/blog/posts/how-to-scrape-threads), and a broader comparison across platforms is in our [roundup of social media scraping tools](https://scrapfly.io/blog/posts/best-social-media-scraping-tools). Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents - [Key Takeaways](#key-takeaways) - [Does Reddit Scraping Still Work in 2026?](#does-reddit-scraping-still-work-in-2026) - [Which Reddit Scraping Method Does This Guide Use?](#which-reddit-scraping-method-does-this-guide-use) - [Which Reddit Data Can You Collect From Public Pages?](#which-reddit-data-can-you-collect-from-public-pages) - [How Do You Set Up the Canonical Python Reddit Scraper?](#how-do-you-set-up-the-canonical-python-reddit-scraper) - [Install Python and scrapfly-sdk for the Reddit Scraper](#install-python-and-scrapfly-sdk-for-the-reddit-scraper) - [Run the scrapfly-scrapers Reddit Tests Before Using the Output](#run-the-scrapfly-scrapers-reddit-tests-before-using-the-output) - [How Do You Scrape Reddit Subreddits and Paginate Posts?](#how-do-you-scrape-reddit-subreddits-and-paginate-posts) - [Parse shreddit-post Fields From a Reddit Subreddit](#parse-shreddit-post-fields-from-a-reddit-subreddit) - [Paginate Reddit Posts With more-posts-cursor](#paginate-reddit-posts-with-more-posts-cursor) - [How Do You Scrape Reddit Posts and Nested Comments?](#how-do-you-scrape-reddit-posts-and-nested-comments) - [Parse Reddit Post Metadata From shreddit-post](#parse-reddit-post-metadata-from-shreddit-post) - [Parse Reddit Comments and Rebuild the Reply Tree](#parse-reddit-comments-and-rebuild-the-reply-tree) - [How Do You Scrape Reddit Profile Posts and Comments?](#how-do-you-scrape-reddit-profile-posts-and-comments) - [Paginate Reddit Profile Posts](#paginate-reddit-profile-posts) - [Parse Reddit Profile Comments](#parse-reddit-profile-comments) - [Why Does Reddit Return 403 or Empty Scraper Output?](#why-does-reddit-return-403-or-empty-scraper-output) - [HTTP 403 or a 200 Challenge Page](#http-403-or-a-200-challenge-page) - [Missing shreddit-post Selectors or Pagination Cursors](#missing-shreddit-post-selectors-or-pagination-cursors) - [Deleted Reddit Authors and Nullable Fields](#deleted-reddit-authors-and-nullable-fields) - [How Do You Keep a Python Reddit Scraper Working?](#how-do-you-keep-a-python-reddit-scraper-working) - [Pin the scrapfly-scrapers Commit and Dependency Versions](#pin-the-scrapfly-scrapers-commit-and-dependency-versions) - [Validate Reddit Output Schemas in CI](#validate-reddit-output-schemas-in-ci) - [FAQ](#faq) - [Summary: Choose a Reddit Access Route and Test It](#summary-choose-a-reddit-access-route-and-test-it) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ http python ### How to Fix 403 Forbidden Errors When Web Scraping Learn why web scrapers get 403 Forbidden errors and how to fix them with 7 Python solutions, from headers to TLS fingerp... ](https://scrapfly.io/blog/posts/403-forbidden-web-scraping) [ python hidden-api ### How to Scrape YouTube in 2026 Learn how to scrape YouTube channel, video, comment, and Shorts data in Python using hidden APIs and yt-dlp. No API key ... ](https://scrapfly.io/blog/posts/how-to-scrape-youtube) [ python scrapeguide ### How to Scrape LinkedIn Profiles, Companies, and Jobs in 2026 LinkedIn aggressively blocks scrapers. This guide shows how to scrape profiles, companies, and jobs anyway using ScrapFl... ](https://scrapfly.io/blog/posts/how-to-scrape-linkedin) ## Related Questions - [ Q Web scraping - what is HTTP 403 status code? ](https://scrapfly.io/blog/answers/403-status-code) - [ Q Python httpx vs requests vs aiohttp - key differences ](https://scrapfly.io/blog/answers/httpx-vs-requests-vs-aiohttp) - [ Q How to handle popup dialogs in Playwright? ](https://scrapfly.io/blog/answers/how-to-click-on-alert-dialog-in-playwright) Scale your web scraping effortlessly, **1,000 free credits** [Start Free](https://scrapfly.io/register)