[Blog](https://scrapfly.io/blog)   /  [python](https://scrapfly.io/blog/tag/python)   /  [How to Scrape Instagram in 2026](https://scrapfly.io/blog/posts/how-to-scrape-instagram)   # How to Scrape Instagram in 2026

 by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) Sep 24, 2026 25 min read [\#python](https://scrapfly.io/blog/tag/python) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) 

 [  ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-instagram "Share on LinkedIn") [  ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-instagram&text=How%20to%20Scrape%20Instagram%20in%202026 "Share on X") [  ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-instagram "Share on Facebook")    

 

 

Summarize this article with

 [  ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-instagram) [  ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-instagram) [  ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-instagram) [  ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-instagram) [  ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-instagram) 



   

Instagram holds valuable data for businesses. You can extract competitor insights, customer sentiment, and market trends from profiles, posts, and comments. However, Instagram makes it hard to scrape their data. In 2026, Instagram's anti-bot defenses have grown more aggressive, mandatory login walls, GraphQL obfuscation, and rapid IP flagging mean that simple scripts with BeautifulSoup or basic HTTP clients no longer work.

In this guide, you'll learn how Instagram blocks scrapers, what data you can extract, and why building your own solution usually fails. We'll also show you a better approach using ScrapFly's maintained Instagram scraper that handles all the blocking challenges automatically.

[How to Scrape Hidden APIsIn this tutorial we'll be taking a look at scraping hidden APIs which are becoming more and more common in modern dynamic websites - what's the best way to scrape them?](https://scrapfly.io/blog/posts/how-to-scrape-hidden-apis)

[**Latest Instagram Scraper Code**github.com/scrapfly/scrapfly-scrapers/instagram-scraper](https://github.com/scrapfly/scrapfly-scrapers/tree/main/instagram-scraper)



## Key Takeaways

- Instagram scraping in 2026 no longer means calling clean JSON endpoints. Profiles and single posts now come from JSON embedded in the page HTML, inside `<script type="application/json" data-sjs>` tags. Only a user's post feed still uses `graphql/query` with a `doc_id`.
- The `web_profile_info` endpoint that most Instagram tutorials still teach returns HTTP 400 with a "make sure you are logged in" message on a growing subset of public profiles, even with no session attached.
- Three defenses kill DIY scrapers: datacenter IPs are blocked on the first request, Python's `requests` and `httpx` are fingerprinted at the TLS layer, and a single residential IP is capped at roughly 200 requests per hour.
- Instagram scrapers usually break silently rather than loudly. The public GraphQL path went down on 2026-06-29 and came back returning null view counts on video posts, so scrapers kept writing rows that were quietly missing fields.
- Only public data is reachable: profiles, posts, reels, comments, and story highlights. No scraper can access private accounts, full follower lists, or current stories, and Instagram's native search sits behind the login wall.
- ScrapFly's open-source Instagram scraper tracks these changes and runs on the [Web Scraping API](https://scrapfly.io/products/web-scraping-api) with anti-bot bypass and residential proxies included, so an Instagram change becomes a `git pull` instead of a re-reverse-engineering project.

**Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime.







## What Instagram Data Can You Scrape?

Instagram's public data provides business intelligence when extracted systematically. Here's what you can scrape and why it matters:

**Profiles** - Extract bio, follower/following counts, verification status, and story highlights. Use case: Build lead lists by scraping verified business profiles in specific niches, then review each bio and highlight reel before outreach.

**Posts** - Capture captions, images, likes, comment counts, timestamps, location tags, and the first page of comments. Use case: Analyze your competitor's top-performing content to understand what resonates with your shared audience and replicate successful formats.

**Reels** - Access captions, likes, and comment counts through the same flow as regular posts. Use case: Track which reel formats in your industry draw the most engagement.

**Comments** - Scrape comment text, timestamps, author profiles, and like counts. Use case: Perform sentiment analysis on competitor posts to identify customer pain points and service gaps you can address.

**Hashtags** - Native hashtag browsing is login-gated as of 2026. Discovery workaround: use Google's `site:instagram.com/p/ keyword` search to find public posts by topic, then scrape those URLs directly. For structured hashtag data, the Instagram Graph API requires Facebook-login authentication. Use case: Find micro-influencers by searching niche keywords via Google and ranking results by engagement rate.

Before you start scraping, it's worth knowing what Instagram restricts at a structural level. Some data is off limits, regardless of which scraper or API you use.



## What You Can't Scrape from Instagram (2026 Limits)

These restrictions apply regardless of which tool you use. They are structural limits, not anti-bot defenses:

- **Private accounts**: no scraper, API, or tool can access them.
- **Full follower and following lists**: behind login the official API returns only `followers_count` not the list itself.
- **Current stories**: login-only. Story highlights are different: they stay on public profiles and remain accessible without logging in.
- **Native hashtag and keyword browsing**: removed from logged-out access in 2024. See "How to Find Instagram Posts Without Native Search" below for the workaround.
- **Contact info beyond the public bio**: only visible if a business profile has made it public.

Understanding these hard limits prevents wasted effort. With the structural ceiling defined, let's look at how Instagram actively blocks access to the data that is public.



## How Instagram Blocks Scrapers (Anti-Bot Detection Explained)

Instagram uses a [multi-layered blocking system](https://scrapfly.io/blog/posts/how-to-know-what-anti-bot-website-uses) designed to identify and block automated scraping. Understanding these systems shows why manual scraping solutions fail and require constant maintenance.

### Rate Limiting &amp; IP Blocking

Instagram enforces strict request quotas to prevent aggressive scraping:

- Request limits: ~200 requests per hour per IP address for non-authenticated users
- Throttling response: After exceeding limits, you receive [HTTP 429 ](https://scrapfly.io/blog/posts/what-is-http-error-429-too-many-requests)
- Block duration: Your IP gets temporarily rate-limited for hours or days depending on violation severity
- Progressive penalties: Repeated violations lead to longer blocks and eventually permanent IP bans

Even if you implement delays and respect rate limits, you're still limited to scraping ~4,800 profiles per day per IP. This is insufficient for serious data collection.

### IP Quality Detection

Instagram analyzes your IP address quality before even processing your request:

- Datacenter IPs blocked instantly: Requests from AWS, DigitalOcean, Google Cloud, and other hosting providers are flagged immediately
- Residential IPs required: Instagram expects requests from genuine consumer ISPs (Comcast, AT&amp;T, etc.)
- ASN reputation checking: Instagram maintains blocklists of ASNs (Autonomous System Numbers) associated with proxies and VPNs
- This runs BEFORE rate limits: A datacenter IP gets blocked on the first request, regardless of how slowly you scrape

This is why you cannot deploy your scraper to a cloud server and expect it to work. Instagram blocks it before you even hit the rate limit.

### Browser Fingerprinting

Instagram analyzes dozens of browser characteristics to detect automation tools:

- TLS/SSL fingerprinting: Python's `requests` library has a unique [TLS handshake signature](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-tls) that Instagram flags as a bot instantly
- HTTP/2 fingerprinting: The order and format of HTTP/2 frames reveals whether you're using a real browser or a scripting library
- Header order consistency: Real browsers send headers in a specific order. Scrapers often randomize or alphabetize them
- [Canvas/WebGL fingerprinting](https://scrapfly.io/blog/posts/browser-fingerprinting-with-creepjs): When JavaScript is enabled, Instagram tests how your browser renders graphics. Automation frameworks produce consistent, detectable signatures

Even if you copy all the correct headers from a real browser, the TLS handshake alone will expose you as a bot within seconds.

### Request Pattern Detection

Instagram's behavioral analysis identifies non-human usage patterns:

- Timing patterns: Perfect 3-second delays between requests look robotic. Humans vary their timing
- Request sequencing: Real users browse naturally (view profile → scroll → click post). Bots often access API endpoints directly without realistic browsing
- Session validation: Instagram expects correlated requests (CSS, images, analytics) alongside your API calls. Scraping only the data endpoints is suspicious
- Cookie behavior: Missing, malformed, or inconsistent cookies signal automation

Instagram's machine learning models are trained on millions of real user sessions. Any deviation from natural human behavior raises red flags.



Instagram's Multi-Layered Anti-Bot Defense SystemEven a well-built scraper gets blocked within hours without serious anti-detection infrastructure. Instagram updates its blocking systems frequently, so scrapers that work today can break by next week.

The detection stack it uses is similar to what [Cloudflare](https://scrapfly.io/blog/posts/how-to-bypass-cloudflare-anti-scraping) and other anti-bot systems apply against automated traffic.



## Instagram Scraper API vs Building Your Own: Which Should You Use?

Once you understand why Instagram blocks scrapers, the real question is who maintains the fix when it breaks. Here is how the four main approaches compare:

| Approach | What it gets you | Rate limit | Maintenance | Cost | Pick when |
|---|---|---|---|---|---|
| DIY Python (`curl_cffi` + residential proxies) | Full control, any public endpoint | ~200 req/hr/IP | Unscheduled fixes: gated endpoints, payload moves, doc\_id changes | Proxies at $1-3/GB | Learning or tiny volume |
| Instaloader (open source) | Profiles and posts, login optional | Same IP limits | Community-maintained | Free | Research, under 1K req/day |
| Official Graph API | Your own Business or Creator accounts only | 200 calls/hr/token | Stable, requires Meta app review | Free | You own the account |
| ScrapFly scraper + Web Scraping API | All public data, anti-bot and proxies handled | Managed | Repository updated for you | Per-request | Production or third-party data |

If you own the account, use the Graph API. For any account you don't own, the public web endpoints are the only path. The real choice is who maintains the scraper when Instagram changes it: you, or someone whose job it is.



## Project Setup

The examples use the Scrapfly Python SDK, plus `nested-lookup` and `jmespath` to walk Instagram's embedded JSON. Install all three with pip:

bash```bash
pip install scrapfly-sdk nested-lookup jmespath
```



With the packages installed, you can run the ready-made Instagram scraper or follow each flow by hand.



## How to Scrape Instagram with ScrapFly (The Easy Way)

ScrapFly's Web Scraping API provides the complete Instagram scraping solution: working scraper code + anti-blocking infrastructure. Clone the repository, configure your API key, and start scraping in 5 minutes.

### What You Get

- Working scraper code: Open source, actively maintained, updated within hours when Instagram changes
- Built-in anti-blocking infrastructure: TLS fingerprinting, header rotation, and behavioral mimicry handled automatically
- Residential proxy network included: 130M+ IPs from real consumer ISPs with no separate proxy bills or configuration
- Automatic updates: When Instagram changes doc\_ids or endpoints, we update the scraper. In 2026 the repo moved single posts to HTML parsing on 2026-06-30, the day after the GraphQL break, and profiles to the embedded payload on 2026-07-30
- Cost control: [Proxy Saver](https://scrapfly.io/products/proxy-saver) serves repeated requests from cache, so a cache hit uses no upstream proxy bandwidth

[Get Started in 5 Minutes](https://scrapfly.io/pricing)

bash```bash
# Clone the scraper repository
git clone https://github.com/scrapfly/scrapfly-scrapers.git
cd scrapfly-scrapers/instagram-scraper

# Configure your ScrapFly API key
export SCRAPFLY_KEY="your_key_here"

# Install dependencies
poetry install

# Start scraping
poetry run python run.py
```



`poetry install` pulls in `nested-lookup` and `jmespath` alongside the Scrapfly SDK. Both packages walk the page's embedded JSON in the examples below.

### How ScrapFly Bypasses Every Defense

Anti-Blocking Bypass: ScrapFly rotates TLS fingerprints to match real Chrome/Firefox browsers, orders HTTP headers correctly, and mimics genuine browser behavior. Instagram sees legitimate browser traffic, not a scraper.

Proxy Management: Our network of [130M+ residential IPs](https://scrapfly.io/blog/posts/best-proxy-providers-for-web-scraping) automatically rotates with each request. Instagram sees requests from real consumer devices across different ISPs and locations, exactly like genuine users.

Rate Limit Handling: Smart throttling and exponential backoff automatically slow down when Instagram pushes back. The scraper adjusts its speed dynamically to stay under the radar.

Proxy Saver: Caches repeated requests and stubs images and CSS, so residential bandwidth goes to the actual API calls.



## How Instagram's Scraping API Works

Instagram doesn't publish a public API, but its web app feeds its front end three different ways. A scraper has to follow whichever path the target data travels on:

1. **JSON embedded in the page HTML** - Instagram ships a large JSON payload inside `<script type="application/json" data-sjs>` tags on the profile and post pages themselves. This is now the path for **profiles** (`xig_user_by_username`) and **single posts** (`xig_polaris_media`). No special headers, no doc\_id. You fetch the page a browser would fetch and read the payload it already contains.
2. **[GraphQL](https://scrapfly.io/blog/posts/web-scraping-graphql-with-python)** - `instagram.com/graphql/query/` with a `doc_id` parameter. Still the path for a **user's post feed** with pagination.
3. **XHR during a rendered session** - **comments** load on scroll, so they are captured from the browser's own network calls rather than requested directly.

As Instagram gates its internal API endpoints, the inline page payload becomes the more reliable target. The logged-out page needs that same data to render at all. Instagram can't gate it without breaking its own page.

### Finding Instagram's Hidden Endpoints

When Instagram updates their platform, endpoints change. Here's how to discover current endpoints when they break:

Step 1: Open Instagram in Chrome/Firefox and open DevTools (F12)

Step 2: Go to Network tab and filter by "Fetch/XHR" to see API calls

Step 3: Browse Instagram normally (visit a profile, view a post, scroll comments)

Step 4: Look for the data in these places, in this order:

- The page source: view it and search for `data-sjs` (profile and post data lives here now)
- `i.instagram.com/api/v1/` (REST endpoints)
- `www.instagram.com/graphql/query` (GraphQL endpoints)

Step 5: Click on an API request to inspect:

- Request payload (for GraphQL, look for `variables` and `doc_id`)
- Response structure (to understand data format)

Page example: When viewing a profile, search the page source for the `xig_user_by_username` key:

text```text
<script type="application/json" data-sjs>{"require":[["ScheduledServerJS","handle",null,[{"__bbox":{...
```



GraphQL example: When loading a user's post grid, you will see a GET request to:

text```text
https://www.instagram.com/graphql/query/?doc_id=9310670392322965&variables=<url-encoded JSON>
```



The `variables` JSON names the target `username` and a page `count`.

### Understanding doc\_id

The common advice when a GraphQL doc\_id breaks is to fall back to `web_profile_info`. That advice is backwards. `web_profile_info` is the path that got gated, and the GraphQL feed path is the one that survived.

The `doc_id` parameter is important for GraphQL scraping but poorly understood. Here's what you need to know:

What is doc\_id?

- Instagram's internal identifier for specific GraphQL query structures
- Maps to a predefined query on Instagram's backend (you cannot define custom queries)
- Example: `doc_id=9310670392322965` retrieves a user's post feed

Why doc\_ids exist:

- Performance: Pre-defined queries are optimized and cached on Instagram's servers
- Security: Prevents custom queries that could overload the database
- Anti-scraping: Changing doc\_ids regularly breaks scrapers

Why doc\_ids change:

- A doc\_id changes when Instagram ships a new version of the query behind it
- The feed's `9310670392322965` has held since February 2025, while the post doc\_id changed twice in early 2025
- No public documentation of current values. You must discover them yourself

How to find current doc\_ids:

1. Open DevTools → Network tab → filter for "graphql"
2. Trigger the action on Instagram (load a user's post grid, scroll for more)
3. Inspect the request payload for `doc_id=` parameter
4. Note the numeric value (e.g., `9310670392322965`)

Which operations still use a doc\_id:

- User post feed: `9310670392322965`
- Single posts: no doc\_id. They come from the page's inline JSON
- Profiles: no doc\_id. They come from the page's inline JSON

The DIY Pain: doc\_id rotation is real, but it wasn't what broke this generation of scrapers. An endpoint got gated and a payload moved into the HTML, so the maintenance burden is wider than watching one number.

ScrapFly Solution: Our open-source repository is updated within hours of Instagram changes. You pull the latest code and keep scraping with no detective work required.

### Required Headers

[Header hygiene](https://scrapfly.io/blog/posts/python-requests-headers-guide) still matters if you hand-roll requests. Headers mattered most when the target was Instagram's internal API endpoints.

Profiles and posts now come from the public page HTML, so the request looks like an ordinary page load. The header surface is much smaller:

python```python
{
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
}
```



Why each header matters:

- x-ig-app-id: identifies the web app to Instagram's internal API endpoints. The current scraper does not need it, because it no longer calls them.
- User-Agent: Must match a real browser signature. Python's default User-Agent gets blocked right away.
- Accept-Language: Instagram tracks inconsistent language preferences across requests. Keep it stable per session.
- Accept-Encoding: Real browsers always accept compression. Omitting this is suspicious.

What happens with wrong headers:

- HTTP 200 with the `data-sjs` payload stripped: your client's TLS fingerprint got flagged. In our 2026-09-23 test, plain `requests` got this page while `curl_cffi` got the full payload
- 400 Bad Request: Malformed headers or missing required fields
- 400 with a "make sure you are logged in" message: the endpoint is gated for that account. That's not a header problem, and fixing headers won't help
- No response: Your IP was flagged and silently dropped



## How to Find Instagram Posts Without Native Search

Instagram removed hashtag and keyword browsing from logged-out access in 2024. Without an account, you cannot use Instagram's search to discover posts or profiles by topic. This creates a discovery gap that no scraper solves on its own.

The workaround is Google. Instagram posts are indexed publicly, so standard `site:` search syntax lets you find post URLs by keyword without touching Instagram directly:

text```text
site:instagram.com/p/ coffee roasters
site:instagram.com/reel/ electric vehicles
```



This returns a list of public post URLs matching your keyword. Feed those URLs into the scraper below to pull captions, engagement metrics, and author profile data.

For scraping the Google results themselves, ScrapFly's [Google search scraper](https://scrapfly.io/blog/posts/how-to-scrape-google) handles pagination and anti-bot detection automatically.

Coverage is not complete. Google's index is not real-time and won't surface every post. Use this method to build a seed list of relevant accounts, then scrape their post history directly once you have their profile URLs.



## How to Scrape Instagram Profiles

Instagram's `web_profile_info` endpoint now returns HTTP 400 with a "make sure you are logged in" message for a growing subset of public profiles, even without a session. The profile page's inline `data-sjs` payload is not gated the same way, so that is where profile data now comes from.

In our own direct test on 2026-09-23, the endpoint answered `google`, `nike`, and `nasa` with HTTP 401 and `"require_login": true`, so the exact status varies.

What you can extract:

- **Full name, username, user ID, and Facebook ID**
- **Bio text and bio links**
- **Follower and following counts**
- **Private, verified, and memorialized flags, plus pronouns and account badges**
- **Threads handle and profile picture URL**
- **Story highlights**: a list of highlights, each with an id, a title, and a cover image URL

**The approach:**Fetch the public profile page, then read the `xig_user_by_username` object from the page's `data-sjs` script tags.

ScrapFly's scraper handles:

- **Anti-bot bypass** so the page loads like a real visitor's
- **Residential proxy rotation to avoid IP blocks**
- **TLS fingerprint matching to bypass blocking detection**

Code snippet from the ScrapFly scraper:

python```python
from scrapfly import ScrapeConfig, ScrapflyClient
from nested_lookup import nested_lookup
import json

scrapfly = ScrapflyClient(key="YOUR_SCRAPFLY_KEY")

BASE_CONFIG = {
    "asp": True,        # Anti Scraping Protection bypass
    "country": "CA",    # change country for relevant results
}

async def scrape_user(username: str):
    """Scrape an Instagram profile from the page's embedded JSON"""
    result = await scrapfly.async_scrape(
        ScrapeConfig(
            url=f"https://www.instagram.com/{username}/",
            **BASE_CONFIG,
        )
    )
    datasets = result.selector.css(
        'script[type="application/json"][data-sjs]::text'
    ).getall()
    hidden = [
        d for d in datasets
        if "xig_user_by_username" in d and "follower_count" in d
    ]
    if not hidden:
        raise ValueError(f"Could not find user data in page: {username}")
    return nested_lookup("xig_user_by_username", json.loads(hidden[0]))[0]

# Example usage
profile = await scrape_user("google")
print(f"Followers: {profile['follower_count']}")
```



The snippet returns the **raw** `xig_user_by_username` object, where the follower field is `follower_count`. The scraper's own `parse_user()` helper renames fields on the way out, so its parsed output calls the same value `followers`.

Here is a trimmed sample of that parsed output for `@google`, from the scraper's `results/user.json`:

json```json
{
  "name": "Google",
  "username": "google",
  "id": "1067259270",
  "followers": 15689074,
  "follows": 45,
  "is_verified": true,
  "highlights": [
    {
      "id": "17989013438800507",
      "title": "Google I/O",
      "cover_image": "https://scontent.cdninstagram.com/..."
    }
  ]
}
```



Key implementation details:

- The **`asp=True`** parameter activates ScrapFly's anti-blocking bypass (TLS fingerprinting, header rotation)
- **Residential proxies** (`country="CA"`) prevent datacenter IP blocks
- Some responses arrive without the `xig_user_by_username` payload, so retry the request when the `ValueError` fires
- Recent posts aren't in this output. Use the post feed flow below to pull a user's posts

The full profile scraper, including `parse_user()`, is in the repository linked at the top of this guide.



Scrapfly

#### Scale your web scraping effortlessly

Scrapfly handles proxies, browsers, and anti-bot bypass — so you can focus on data.

[Try Free →](https://scrapfly.io/register)## How to Scrape Instagram Posts

Post data includes captions, media URLs, engagement metrics, and comments. The public GraphQL path for single posts broke on 2026-06-29, and the scraper moved to HTML parsing on 2026-06-30.

A maintainer-tracked bug report, [instagrapi issue #2713](https://github.com/subzeroid/instagrapi/issues/2713), recorded the break. It reads: "the public GQL stopped working yesterday, and it was impossible to retrieve any data from it."

When the path came back, video posts were no longer returning view counts. That's a silent partial break: the scraper runs, but a field goes missing.

What you can extract:

- **Post ID, shortcode, and timestamp**
- **Image URL and carousel images**
- **Caption text**
- **Like and comment counts**
- **First page of comments**

**The approach:**Fetch the post page, find the `data-sjs` script containing `xig_polaris_media`, and walk into it.

ScrapFly's scraper handles:

- **Anti-bot bypass and residential proxies**, so the page loads like a real visitor's
- **The `if_not_gated_logged_out` key**: logged-out requests get the payload nested under it, so the parser checks there first
- **Different post types**: photos, videos, reels, carousels

Code snippet from the ScrapFly scraper:

python```python
import json
import re

def extract_polaris_media(html: str):
    """Find the xig_polaris_media payload inside the page's data-sjs scripts"""
    scripts = re.findall(r'<script[^>]*data-sjs[^>]*>(.*?)</script>', html, re.DOTALL)
    for script in scripts:
        if "xig_polaris_media" not in script:
            continue
        try:
            modules = json.loads(script)["require"][0][3][0]["__bbox"]["require"]
        except (KeyError, IndexError, TypeError, json.JSONDecodeError):
            continue
        for module in modules:
            if not isinstance(module, list) or len(module) <= 3:
                continue
            for entry in module[3]:
                if not isinstance(entry, dict):
                    continue
                media = (
                    entry.get("__bbox", {}).get("result", {})
                    .get("data", {}).get("xig_polaris_media")
                )
                if media:
                    return media
    return None

async def scrape_post(url_or_shortcode: str):
    """Scrape a single Instagram post by parsing the HTML page"""
    if "http" not in url_or_shortcode:
        url = f"https://www.instagram.com/p/{url_or_shortcode}/"
    else:
        url = url_or_shortcode

    result = await scrapfly.async_scrape(ScrapeConfig(url=url, **BASE_CONFIG))
    media = extract_polaris_media(result.content)
    if not media:
        raise ValueError(f"Could not find post data in page: {url}")
    # logged-out pages nest the real payload one level deeper
    return media.get("if_not_gated_logged_out") or media
```



Logged-out requests get the post payload nested under `if_not_gated_logged_out`, so the parser checks for that key before falling through to the top level.

A trimmed sample of the raw post object for `Cs9iEotsiGY`, from a live run on 2026-09-23:

json```json
{
  "pk": "3115796369385857432",
  "code": "Cs9iEotsiGY",
  "like_count": 5110,
  "comment_count": 99,
  "taken_at": 1685651910,
  "caption": {"text": "It's a match made in Google Lens"}
}
```



Key implementation details:

- The **shortcode** is the unique post identifier (e.g., `Cs9iEotsiGY` from URL `/p/Cs9iEotsiGY/`)
- The caption sits under **`caption.text`** and the comment total under **`comment_count`**
- The scraper's `parse_post()` helper also returns a **`comments`** list with the first page of comments

### Scraping a User's Post Feed

The user post feed is the only flow that still uses a doc\_id. Send a `GET` request to `instagram.com/graphql/query/` with `doc_id=9310670392322965` and a `variables` object naming the username and page size.

The response nests posts under `data.xdt_api__v1__feed__user_timeline_graphql_connection`, and `page_info.has_next_page` tells you when to stop paginating.

The full post and post feed scrapers are in the repository linked at the top of this guide.



## How to Scrape Instagram Comments

Comments provide sentiment data, user engagement patterns, and conversation threads. The comment scraper was reworked on 2026-06-11. Comments now load as the reader scrolls, so the scraper renders the page and harvests Instagram's own network calls.

What you can extract:

- **Comment text and timestamp**
- **Commenter username, profile, verification status**
- **Like counts per comment**

**The approach:**The scraper renders the post page with `render_js=True` on the residential pool. A `js_scenario` waits for the comment icon, dismisses any dialog, then scrolls the comment container repeatedly.

The scraper then reads `browser_data.xhr_call` from the result. It keeps the response bodies that contain `comments_connection`.

This is a different class of technique from the rest of this guide. Instead of guessing which endpoint holds the data, you let Instagram's front end make the call and read the answer. That's why it survives endpoint churn.

ScrapFly's scraper handles:

- **Browser rendering with residential proxies**
- **The scroll scenario** that triggers comment loading
- **De-duplication by comment `id`**, which is required because scroll-triggered calls overlap

Code snippet from the ScrapFly scraper:

python```python
async def scrape_post_comments(url: str):
    """Scrape comments by rendering the post page and capturing its XHR calls"""
    result = await scrapfly.async_scrape(
        ScrapeConfig(
            url=url,
            render_js=True,
            proxy_pool="public_residential_pool",
            js_scenario=COMMENTS_JS_SCENARIO,   # scrolls the comment container
            **BASE_CONFIG,
        )
    )

    comments, seen = [], set()
    for xhr in result.scrape_result.get("browser_data", {}).get("xhr_call") or []:
        body = (xhr.get("response") or {}).get("body")
        if not body or "comments_connection" not in body:
            continue
        data = json.loads(body)
        edges = jmespath.search(
            "data.xig_polaris_media.comments_connection.edges[].node", data
        ) or []
        for node in edges:
            if node["id"] not in seen:
                seen.add(node["id"])
                comments.append(node)
    return comments
```



`COMMENTS_JS_SCENARIO` is the scroll script. Read the full script in the scraper's [instagram.py](https://github.com/scrapfly/scrapfly-scrapers/blob/main/instagram-scraper/instagram.py).

A trimmed sample of one raw comment node, from a live run on 2026-09-23:

json```json
{
  "pk": "17988940742023070",
  "text": "It is not working in India",
  "created_at": 1685373846,
  "comment_like_count": 0,
  "user": {"username": "gs_4584", "is_verified": false}
}
```



Key implementation details:

- There is **no manual cursor**. The scroll scenario surfaces comments as they load
- **De-duplication is required**, because the same comment can arrive in more than one XHR call
- In our 2026-09-23 runs, some renders captured no comments at all, so retry when the list comes back empty



## How to Tell When Your Instagram Scraper Has Broken

Instagram breaks are usually partial, not total. The scraper keeps writing rows while fields quietly go null, and nothing raises an error.

Here are the per-flow failure signatures and what they usually mean:

| Symptom | What it usually means | Where to look |
|---|---|---|
| HTTP 400 with a "make sure you are logged in" message | Instagram refused a logged-out request from that IP. A 401 variant says "Please wait a few minutes", so the status alone doesn't separate gating from throttling | Retry from a different residential IP. For a stable path, switch to the page's inline `data-sjs` payload |
| `ValueError: Could not find user data in page` | The inline payload key was renamed, or the page returned a login wall instead of the profile | Search the raw HTML for `data-sjs` and check which keys are present |
| Rows still arriving, but a field is uniformly null | A partial upstream break. This is the dangerous one. | Compare a known-good record. Alert on null rate per field, not on error rate |
| HTTP 401 on some URLs but not others | Per-content gating rather than a global break | Test a second post from a different account before concluding the flow is dead |
| Comments return empty with no error | The scroll scenario did not fire, or the XHR pattern changed | Check whether any captured XHR body contains `comments_connection` |

Assert on field presence rather than on HTTP status, since a 200 response can still carry a gutted payload. Record a last-success timestamp per flow, and alert on null-rate drift.

The [instagrapi #2713](https://github.com/subzeroid/instagrapi/issues/2713) report shows why. The reporter's scraper recovered from a total outage into a state where video posts silently returned no view counts.

The fix for both 2026 breaks was already in ScrapFly's repository before most readers noticed. That's the difference between a `git pull` and a debugging week.



## How to Scrape Instagram with Proxies

Proxies are mandatory for Instagram scraping at any scale. Instagram's IP quality detection blocks datacenter IPs instantly, and rate limits force you to rotate residential IPs to maintain scraping speed.

### Best Proxies for Instagram Scraping (Residential vs Datacenter)

**Datacenter Proxies: Do not use**

- **Blocked instantly** by Instagram's IP quality checks
- **No request volume possible**. Banned on first request
- Cheaper per GB, but **100% failure rate** makes cost irrelevant

**Residential Proxies: Required**

- **IPs from real consumer ISPs** (Comcast, Verizon, AT&amp;T, etc.)
- **Pass Instagram's IP quality detection**
- Each IP allows **~200 requests/hour** before rate limiting
- **Geographic targeting** (e.g., US-only IPs for US-focused scraping)

**Mobile Proxies: Premium Option**

- **IPs from mobile carriers** (4G/5G networks)
- **Highest trust score**. Instagram rarely blocks mobile IPs
- **Better rate limits** (~300 requests/hour per IP)
- **More expensive** ($60-120/month per IP vs $1-3 for residential)

Recommendation: [Residential proxies](https://scrapfly.io/blog/posts/introduction-to-proxies-in-web-scraping) are the sweet spot for Instagram scraping. Mobile proxies offer marginal improvement at 10-20x the cost. Not worth it unless you are scraping millions of profiles daily.

[How to Rotate Proxies in Web ScrapingIn this article we explore proxy rotation. How does it affect web scraping success and blocking rates and how can we smartly distribute our traffic through a pool of proxies for the best results.](https://scrapfly.io/blog/posts/how-to-rotate-proxies-in-web-scraping)

### How to Rotate Proxies for Instagram Scraping

Proxy rotation strategies determine your scraping speed and block rate:

**Sticky Sessions (Recommended):**

- Use the **same IP for 5-10 minutes**, then rotate
- **Mimics real user behavior** (one person does not change IPs every 10 seconds)
- Allows **~15-30 requests per IP** before rotation
- Instagram's behavioral analysis **flags instant IP changes** as suspicious

**Request-Level Rotation (Aggressive):**

- **New IP for every single request**
- **Maximizes speed** but looks unnatural to Instagram
- **Higher block rate**. Use only with anti-bot bypass (like ScrapFly)
- **Necessary when scraping 10,000+ profiles/hour**

**Smart Rotation Based on Response:**

- **Rotate immediately** on 429 (rate limit) or 403 (block)
- **Continue using same IP** while responses are 200 OK
- **Implements exponential backoff**: 2s delay → 4s → 8s → 16s before rotating
- **Reduces wasted proxy bandwidth**

**ScrapFly's automatic proxy management:**

- **Automatic rotation** using sticky sessions by default
- **Instant rotation** on rate limits or blocks
- **Geographic pinning** (keep requests in same country/region)
- **Proxy pool health monitoring** (removes dead IPs automatically)

### Instagram Proxy Costs

Residential proxies are billed per GB of bandwidth consumed. Here's what Instagram scraping costs:

**Data usage per request type:**

- **Profile scrape**: ~50-100 KB per profile
- **Post scrape**: ~30-80 KB per post
- **Comment scrape**: ~20-50 KB per comment page

**Example scraping job: 10,000 Instagram profiles**

- 10,000 profiles × 75 KB average = **750 MB**
- Standard residential proxy cost: **$10-15 per GB**
- **Total cost: $7.50-11.25** in proxy bandwidth

**ScrapFly's Proxy Saver feature:**

- **Caches static content** (profile images, CSS, JavaScript)
- **Only uses residential bandwidth** for actual API calls
- **Serves cache hits with no upstream bandwidth**, so repeat scrapes cost less



## Scale Instagram Scraping with the Web Scraping API

Everything above works without ScrapFly, but keeping it working long-term means tracking gated endpoints, rotating TLS fingerprints, and maintaining a residential proxy pool yourself.



ScrapFly's [Web Scraping API](https://scrapfly.io/products/web-scraping-api) is a single HTTP endpoint for collecting web data at scale, with a **99.99% success rate** across **130M+ proxies in 190+ countries**.

- [Anti-Scraping Protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - automatically defeats Cloudflare, DataDome, PerimeterX, Akamai, and 90+ other bot systems.
- [Smart proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) - residential and datacenter pools with country and ASN level geo-targeting.
- [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - render SPAs and dynamic pages through real cloud browsers.
- [Browser automation scenarios](https://scrapfly.io/docs/scrape-api/javascript-scenario) - scroll, click, fill forms, and wait for elements without managing a browser fleet.
- [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - return pages as HTML, JSON, clean text, or LLM ready Markdown.
- [Session management](https://scrapfly.io/docs/scrape-api/session) - keep cookies, headers, and IPs consistent across multi step flows.
- [Smart caching](https://scrapfly.io/docs/scrape-api/getting-started#api_param_cache) - cache successful responses to cut cost on repeat scraping jobs.
- [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Scrapy](https://scrapfly.io/docs/sdk/scrapy), and [no-code integrations](https://scrapfly.io/docs/integration/getting-started) including [Make](https://scrapfly.io/integration/make), [n8n](https://scrapfly.io/integration/n8n), [Zapier](https://scrapfly.io/integration/zapier), [LangChain](https://scrapfly.io/integration/langchain), and [LlamaIndex](https://scrapfly.io/integration/llamaindex).

Every code sample in this guide runs on this API, from the profile scraper to the comment renderer.



### Web Scraping API

Scrape any website with our powerful API. Anti-bot bypass, JavaScript rendering, and rotating proxies built-in.



[Try Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started)



## FAQ

How to get Instagram user ID from username?Fetch the profile page with the `scrape_user()` function from the profile section and read the `pk` field from the raw `xig_user_by_username` object. Don't call `web_profile_info` for this, since that endpoint is gated and plain `httpx` gets TLS-fingerprinted anyway.

python```python
profile = await scrape_user("google")
print(profile["pk"])  # Output: 1067259270
```



The `pk` field is the numeric user ID. It's the same value `parse_user()` renames to `id`.







How to get Instagram username from user ID?Use Instagram's **public mobile API endpoint**:

python```python
import httpx

iphone_api = "https://i.instagram.com/api/v1/users/{}/info/"
iphone_user_agent = "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Mobile/14G60 Instagram 12.0.0.16.90"

response = httpx.get(
    iphone_api.format("1067259270"),
    headers={"User-Agent": iphone_user_agent}
)
username = response.json()['user']['username']
print(username)  # Output: google
```



The iPhone user agent makes the request look like it came from Instagram's own app.







How do I handle Instagram's rate limiting when scraping at scale?Instagram rate limiting requires a **three-part strategy**:

1. **Residential proxy rotation:** Use **50-100+ residential IPs** and rotate them in sticky sessions (**5-10 minutes per IP**). Each IP allows **~200 requests/hour**.
2. **Realistic delays:** Space requests **2-5 seconds apart** with random variance. Perfect timing intervals look robotic.
3. **Exponential backoff:** When you receive a **429 error**, back off exponentially (wait 2s, then 4s, then 8s, etc.) before retrying.

ScrapFly handles all three automatically, you specify your desired scraping speed and we manage rate limits, retries, and proxy rotation.







Can I scrape Instagram stories or reels data?Current stories are not accessible without login. Story highlights are, and the profile scraper returns them with an id, a title, and a cover image for each. Reels are accessible as posts through the post flow.







How do I extract Instagram comments and engagement metrics?Render the post page and capture its XHR calls, then read comments from `comments_connection.edges[].node` and de-duplicate them by `id`. Engagement metrics sit on the post object as `like_count` and `comment_count`.







What are the most common Instagram scraping challenges?- **Silent breaks** - endpoints get gated, payloads move into the page HTML, and fields go null without an error.
- **IP blocks** - datacenter IPs are banned instantly. Use residential proxies with rotation.
- **TLS fingerprinting** - Python libraries have detectable signatures. Use a tool like ScrapFly that rotates fingerprints.
- **Rate limits** (200 req/hour per IP) - rotate across residential IPs with sticky sessions.
- **Behavioral detection** - add random delays and mimic realistic browsing sequences.







Does Instagram's `web\_profile\_info` endpoint still work?It returns HTTP 400 with a "make sure you are logged in" message on a growing subset of public profiles, even without a session. Profile data now comes from the inline `data-sjs` payload on the profile page.







Do I still need the `x-ig-app-id` header?The header identifies the web app to Instagram's internal API endpoints. The current scraper doesn't send it, because it fetches public pages rather than those endpoints.







Why does my Python Instagram scraper get 403 errors immediately?TLS fingerprinting. Python's `requests` and `httpx` have handshake signatures Instagram flags, and you often get a 200 page with the data stripped rather than a 403.

Your options are browser automation (real fingerprints, but slow), [curl\_cffi](https://scrapfly.io/blog/posts/curl-impersonate-scrape-chrome-firefox-tls-http2-fingerprint) (mimics Chrome's TLS), or ScrapFly (rotates TLS fingerprints automatically).







Is scraping Instagram data legal?US courts have generally allowed scraping public, logged-out data (Meta v. Bright Data, 2024, hiQ v. LinkedIn). Login-walled or private content is a different matter and violates Instagram's ToS. EU data collection triggers GDPR regardless of where you operate. This is not legal advice.







Does Instagram have an official API in 2026?







## Related Guides

- [Scrape TikTok](https://scrapfly.io/blog/posts/how-to-scrape-tiktok-python-json) - Extract profiles, videos, and comments from TikTok using similar anti-bot bypass techniques
- [Scrape Facebook Marketplace](https://scrapfly.io/blog/posts/how-to-scrape-facebook) - Handle Facebook's authentication and anti-bot measures for marketplace and event data
- [Scrape X.com (Twitter)](https://scrapfly.io/blog/posts/how-to-scrape-twitter) - Access Twitter's GraphQL API for posts, profiles, and real-time data
- [Social Media Scraping Guide](https://scrapfly.io/blog/posts/social-media-scraping) - Broad overview of scraping strategies across all major social platforms



## Summary

Instagram scraping in 2026 requires working through complex blocking systems: IP quality detection, TLS fingerprinting, rate limiting, and behavioral analysis. Building a scraper from scratch means constant maintenance, because Instagram gates endpoints and moves payloads without warning.

We covered the three paths Instagram's data now travels. Page-embedded JSON carries profiles and single posts, GraphQL with a `doc_id` carries the post feed, and captured XHR calls carry comments.

Two dated 2026 breaks show why that matters: the GraphQL outage on 2026-06-29, and the `web_profile_info` gating the scraper moved off on 2026-07-30.

The smart approach: Start with ScrapFly's working Instagram scraper that includes anti-blocking bypass, residential proxies, and automatic updates when Instagram changes. This saves you hundreds of hours in maintenance and debugging.



Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect:

- Do not scrape at rates that could damage the website.
- Do not scrape data that's not available publicly.
- Do not store PII of EU citizens protected by GDPR.
- Do not repurpose *entire* public datasets which can be illegal in some countries.

Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer.

 

   [  Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents















 

  Table of Contents- [Key Takeaways](#key-takeaways)
- [What Instagram Data Can You Scrape?](#what-instagram-data-can-you-scrape)
- [What You Can't Scrape from Instagram (2026 Limits)](#what-you-can-t-scrape-from-instagram-2026-limits)
- [How Instagram Blocks Scrapers (Anti-Bot Detection Explained)](#how-instagram-blocks-scrapers-anti-bot-detection-explained)
- [Instagram Scraper API vs Building Your Own: Which Should You Use?](#instagram-scraper-api-vs-building-your-own-which-should-you-use)
- [Project Setup](#project-setup)
- [How to Scrape Instagram with ScrapFly (The Easy Way)](#how-to-scrape-instagram-with-scrapfly-the-easy-way)
- [How Instagram's Scraping API Works](#how-instagram-s-scraping-api-works)
- [How to Find Instagram Posts Without Native Search](#how-to-find-instagram-posts-without-native-search)
- [How to Scrape Instagram Profiles](#how-to-scrape-instagram-profiles)
- [How to Scrape Instagram Posts](#how-to-scrape-instagram-posts)
- [How to Scrape Instagram Comments](#how-to-scrape-instagram-comments)
- [How to Tell When Your Instagram Scraper Has Broken](#how-to-tell-when-your-instagram-scraper-has-broken)
- [How to Scrape Instagram with Proxies](#how-to-scrape-instagram-with-proxies)
- [Scale Instagram Scraping with the Web Scraping API](#scale-instagram-scraping-with-the-web-scraping-api)
- [FAQ](#faq)
- [Related Guides](#related-guides)
- [Summary](#summary)
 
    Join the Newsletter  Get monthly web scraping insights 

 

  



Scale Your Web Scraping

Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits.

  No credit card required  1,000 free API credits  Anti-bot bypass included 

 [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) 

 Not ready? Get our newsletter instead. 

 

 ## Related Articles

 [  

 python data-parsing 

### Web Scraping Emails using Python

In this tutorial we'll take a look at email scraping. How to crawl pages and extract email addresses using Python and wh...

 

 ](https://scrapfly.io/blog/posts/how-to-scrape-emails-using-python) [  

 python hidden-api 

### How to Scrape YouTube in 2026

Learn how to scrape YouTube channel, video, comment, and Shorts data in Python using hidden APIs and yt-dlp. No API key ...

 

 ](https://scrapfly.io/blog/posts/how-to-scrape-youtube) [  

 http python 

### How to Effectively Use User Agents for Web Scraping

In this article, we’ll take a look at the User-Agent header, what it is and how to use it in web scraping. We'll also ge...

 

 ](https://scrapfly.io/blog/posts/user-agent-header-in-web-scraping) 

  



   



 Scale your web scraping effortlessly, **1,000 free credits** [Start Free](https://scrapfly.io/register)