[Blog](https://scrapfly.io/blog) / [python](https://scrapfly.io/blog/tag/python) / [How to Scrape Twitter (X.com) with Python in 2026](https://scrapfly.io/blog/posts/how-to-scrape-twitter) # How to Scrape Twitter (X.com) with Python in 2026 by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) Jul 02, 2026 19 min read [\#python](https://scrapfly.io/blog/tag/python) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-twitter "Share on LinkedIn") Every few months, X.com rotates the internals scrapers depend on: new doc\_ids, guest-token rules, tighter rate limits. Another batch of GitHub scrapers quietly dies. The free stack, snscrape, Twint, and Nitter, is now a graveyard. This guide covers what still works in 2026. You'll see how X.com's hidden GraphQL API operates, the four real ways to get data and their honest trade-offs, and working Python code for scraping profiles and tweets. [**Latest X.com Scraper Code**github.com/scrapfly/scrapfly-scrapers/twitter-scraper](https://github.com/scrapfly/scrapfly-scrapers/tree/main/twitter-scraper) ## Key Takeaways - **Twitter's official API is priced out for most projects.** No free read tier remains. - **The free-library era is over.** snscrape, Twint, and Nitter all broke on guest lockdown. - **X.com's API hides behind three rotating guards:** guest tokens, doc\_ids, rate limits. - **Public data only:** profiles and tweets, not DMs, protected accounts, or follower lists. - **The real cost of Twitter data is maintenance, not code, and it never stops.** - **Scrapfly's maintained scraper absorbs X.com's changes,** so your pipeline survives. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## Can You Still Scrape Twitter in 2026? Yes. Public Twitter data is still scrapeable in 2026 without the official API. But every easy path has closed: the free libraries are dead, anonymous browsing is limited, and X.com actively rotates its internals. What changed comes down to three shifts. X.com locked down anonymous guest access, priced its API out of reach, and tightened bot detection on every public page. Each one gets a full treatment later in this guide. The data is still worth the effort. Teams scrape X.com for sentiment and market research, brand monitoring, and machine-learning datasets built from public posts. Beginners feel the pain first. In an [r/Python thread](https://www.reddit.com/r/Python/comments/1nblyt6/webscraping_twitter_or_any/) (Sep 2025), a developer followed a popular snscrape tutorial and found it dead. They then hit the official API rate limit "after one try." The top reply: scraping Twitter is "getting harder by the day." If you scrape across platforms, the same lockdown story is playing out across the wider social media scraping scene. [Social Media Scraping in 2026Compare four scraping methods across seven platforms. Difficulty ratings, anti-bot techniques, and Python examples for Instagram, Twitter/X, TikTok, LinkedIn, YouTube, Facebook, and Threads.](https://scrapfly.io/blog/posts/social-media-scraping) Next, let's draw the line between what you can pull and what stays locked. ## What Twitter Data Can You Scrape (and What's Off-Limits)? Without logging in, you can scrape public profiles, individual tweets and their visible replies, and embedded media metadata. You can't scrape protected accounts, DMs, follower lists, or full search results. Those sit behind authentication. Here's the boundary at a glance: | Scrapeable without login | Login-gated | |---|---| | Public profile fields (bio, counts, verification) | Protected and suspended accounts | | Individual tweets and their metadata | Direct messages | | Public engagement counts (likes, retweets, views) | Full follower and following lists | | Embedded media URLs | Keyword and timeline search results | Anonymous access also comes with limits beyond the table. Reply depth is shallow, and rate-limited anonymous views cut off long timelines before you reach the end. This guide only covers the public side. The login-gated surface needs a real account, which carries suspension risk, and the legal section below explains why that line matters. First, the question every developer asks before writing code. ## Is It Legal to Scrape Twitter (X)? Scraping public Twitter data is generally legal in the US. The hiQ v. LinkedIn ruling found that accessing public web data doesn't violate the Computer Fraud and Abuse Act (CFAA). But it can still breach X's Terms of Service, a contract issue, not a crime. That distinction trips up a lot of people. A Terms of Service violation can get your account or IP banned and exposes you to civil claims. A CFAA violation involves accessing non-public data, and that's the line you don't cross. Three rules keep public scraping on safe ground: - **Public data only.** Don't touch protected accounts, DMs, or anything behind a login. - **Don't hoard.** Store scraped data no longer than your use case needs. - **Respect rate limits.** Overloading servers is where legal risk gets real. EU readers have one more layer. Public posts can still contain personal data, so GDPR applies when you store or process it. As [ScrapeGraphAI](https://scrapegraphai.com/blog/tweet-scraper) puts it (May 2026), the safe move is to stick to public data only. This section is informational, not legal advice. With the boundaries clear, here are the four ways developers get Twitter data today. ## 4 Ways to Get Twitter Data in 2026 There are exactly four working ways to get Twitter data in 2026. Pay for the official API, resurrect a free library (spoiler: you mostly can't), build and maintain your own scraper, or use a maintained scraper API. Here's the honest comparison. | | Official X API | Free libraries | DIY browser automation | Maintained scraper API | |---|---|---|---|---| | **Cost** | High (paid tiers only) | Free | Proxy + dev time | Per-request pricing | | **Setup time** | Low | Low (when they work) | High | Low | | **Maintenance** | None | Dead or breaks weekly | 10-15 hrs/month | Outsourced | | **Data coverage** | Full (within caps) | Near zero now | Public pages | Public pages | | **Ban risk** | None (compliant) | High | High | Low | No single option wins for everyone. The right choice depends on your volume, your budget, and how much downtime you can tolerate. ### Option 1: The Official X API (Priced Out for Most) The official X API works and is the compliant choice, if your budget survives it. There's no free read tier anymore, so any real project pays. Pricing splits into pay-per-use for new developers and legacy subscription tiers. Pay-per-use bills per post read with a monthly cap in the low millions. The legacy Basic tier runs a few hundred dollars a month for tight read caps. Pro jumps to several thousand, and Enterprise starts in the tens of thousands. Check the [official X API docs](https://docs.x.com/x-api/introduction) for current numbers. The standard Python client is [Tweepy](https://www.tweepy.org/), which wraps authentication and endpoints in a few lines. The API is the right call for low volume, compliance-sensitive work, or posting bots. For scraping public pages at scale, the cost makes it a non-starter. ### Option 2: Free Scraper Libraries: snscrape, Twint, and Nitter (RIP) The free Twitter scraping stack is a graveyard. Twint was abandoned years ago, snscrape broke when X killed anonymous guest access in 2023, and the Nitter instance network collapsed shortly after. Here's what each one did and why it died: - **Twint:** Pulled tweets without the API by scraping the old web frontend. It stopped working once X retired those endpoints. - **snscrape:** Read public timelines through unofficial endpoints. It broke when X required guest tokens for those calls. - **Nitter:** A privacy frontend that proxied public tweets. Its public instances shut down after X blocked the guest accounts they ran on. They died together because they shared one dependency: anonymous guest access, the exact mechanism X locked down. The Guest Tokens section below explains the lock itself. What's left is semi-working, not free. Session-based libraries like tweety-ns and Scweet run on your own logged-in account. They function, but they're account-ban roulette, as practitioners noted in an [r/webscraping thread](https://www.reddit.com/r/webscraping/comments/1drukk9/scraping_xtwitter_tweet_ids/) (Jun 2024). Beginners still land on dead snscrape tutorials first, which is exactly where that r/Python developer started. ### Option 3: Build Your Own with Playwright or Selenium DIY Twitter scraping that works in 2026 means browser automation. That's [Web Scraping with Playwright and Python](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python) or [Web Scraping with Selenium and Python](https://scrapfly.io/blog/posts/web-scraping-with-selenium-and-python) with residential proxies and careful session management. It also means a standing commitment to fix the scraper whenever X.com changes. Three things break constantly on a DIY scraper: - **Guest tokens expire.** Every GraphQL call needs one, they last a few hours, and acquisition methods shift every few weeks. - **doc\_ids rotate.** The identifiers that route GraphQL queries change every 2-4 weeks, and you track 8-12 of them at once. - **IP blocks land fast.** Datacenter IPs get flagged within a request or two, so you need residential proxies in rotation. The cost isn't writing the scraper. It's re-fixing it every few weeks, which typically eats 10-15 hours a month. For a one-engineer team, that becomes a permanent side job. A real DIY scraper starts with a browser session pointed at X.com through a proxy: python```python from playwright.sync_api import sync_playwright # Conceptual skeleton, not a maintained solution with sync_playwright() as pw: browser = pw.chromium.launch( proxy={"server": "http://residential-proxy:8000"} ) page = browser.new_page() page.goto("https://x.com/x") page.wait_for_selector("[data-testid='primaryColumn']") # From here you still handle guest tokens, doc_ids, and rate limits yourself print(page.content()) ``` This skeleton opens the page. It does nothing to solve token rotation, doc\_id tracking, or blocking, which is where the real work lives. ### Option 4: A Maintained Twitter Scraper API The fourth option outsources maintenance. A scraper API tracks X.com's changes, so your pipeline doesn't break when guest tokens or doc\_ids rotate. The trade-off: it costs money and adds a dependency, but wins when uptime matters more than DIY control. Scrapfly publishes an open-source [twitter-scraper](https://github.com/scrapfly/scrapfly-scrapers/tree/main/twitter-scraper) you can clone and run today. When X.com changes its defenses, the maintained scraper and the underlying [Web Scraping API](https://scrapfly.io/web-scraping-api) absorb the change. Your code keeps returning data, and the anti-bot bypass and residential proxy rotation stay handled for you. That covers the decision. The rest of this guide goes deep on the mechanics no free tool could survive. Then it shows working code for the two surfaces you can scrape. Scrapfly #### Scale your web scraping effortlessly Scrapfly handles proxies, browsers, and anti-bot bypass — so you can focus on data. [Try Free →](https://scrapfly.io/register)## How Does Twitter's Hidden API Work? X.com's web app is a GraphQL client. Every profile and timeline you see in the browser is fetched from a hidden internal API. Scraping X.com means speaking that API's language: guest tokens, doc\_ids, and rate limits. The request flow is consistent: 1. The browser loads an X.com page. 2. JavaScript initializes and requests a guest token. 3. X.com returns the token, valid for a few hours. 4. Page JavaScript makes GraphQL queries using the token. 5. Each query carries a doc\_id that names the backend operation. 6. The backend returns JSON, and the frontend renders it. Without a guest token, you can't make queries. Without the right doc\_id, your query matches no operation. Without residential proxies and pacing, X.com blocks your IP. Each piece deserves a closer look. ### Guest Tokens: Twitter's Session Gatekeeper A guest token is a temporary credential that proves a request comes from a real session, not a raw bot. Every call to X.com's GraphQL backend needs one. Tokens expire within a few hours to limit reuse and abuse. X.com also ties each token to the IP that requested it, so rotating IPs mid-session breaks the token. When it expires, your scraper stops cold until it acquires a new one. This is the mechanism that killed the free libraries. They all assumed anonymous access that no longer exists. ### doc\_ids: Why Scrapers Break Every Few Weeks A doc\_id is a unique identifier for a GraphQL operation. Each query type, fetch profile, get tweet, load timeline, has its own. They carry no public documentation and look like random strings. X.com rotates doc\_ids every 2-4 weeks to break reverse-engineered scrapers. A working scraper tracks 8-12 of them at once, across user, tweet, and timeline queries. When they rotate, hardcoded queries fail silently or return empty results. This rotation is what broke snscrape and the rest. A browser-driven scraper sidesteps it by letting the page request its own current doc\_ids, which is the approach the code below uses. ### What Changed Between 2023 and 2026 X.com has shipped defensive changes roughly every 2-4 weeks since 2023, each one tightening the screws on scrapers. The timeline below is a factual record of what broke and when. - **February 2023:** Free API access ends; paid tiers introduced with steep cost barriers. - **March 2023:** New rate limits cut free-tier usage; many apps stopped working. - **June 2023:** Guest token acquisition changed; existing scrapers broke. snscrape and Nitter start failing. - **August 2023:** Rate limits dropped from 450 to 300 requests per hour; datacenter IP blocking increased. - **November 2023:** GraphQL changes forced doc\_id updates across all query types. - **January 2024:** Guest token format and expiry changed; [How TLS Fingerprint is Used to Block Web Scrapers?](https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-tls) checks tightened. - **April 2024:** doc\_ids rotated again; anti-scraping response headers added. - **July 2024:** Cookie validation changed; session handling got stricter. - **October 2024:** IP reputation scoring tightened; rotating proxies flagged earlier. - **January 2025:** Guest tokens bound to browser fingerprints; datacenter IPs permanently banned. - **2026:** Stricter rate limits and tighter token validation across the board. This isn't a stable target. The next two sections show how to scrape it anyway. ## Project Setup The working code uses the [scrapfly-sdk](https://pypi.org/project/scrapfly-sdk/) package. It handles guest tokens, proxies, and the browser render in one request. Install it with pip: bash```bash pip install scrapfly-sdk ``` The scraper sets two flags on every request: `asp=True` for anti-scraping bypass and `render_js=True` to load the GraphQL responses in a real browser. With that in place, let's scrape a profile. ## How to Scrape Twitter Profiles with Python You scrape a Twitter profile by loading the user page in a real browser and capturing the GraphQL response it fires in the background. The browser requests its own current doc\_ids, so you never track them by hand. **Step 1: Configure the browser request.** Set the base config once, with anti-scraping protection and JavaScript rendering enabled. python```python import os import json from scrapfly import ScrapeConfig, ScrapflyClient SCRAPFLY = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"]) BASE_CONFIG = { "asp": True, # anti-scraping protection bypass "render_js": True, # X.com is a JavaScript app, so render it } ``` **Step 2: Load the page and capture the background request.** Wait for the `UserTweets` call, which carries the profile JSON. python```python async def scrape_profile(url: str) -> dict: result = await SCRAPFLY.async_scrape(ScrapeConfig( url, auto_scroll=True, wait_for_selector="xhr:UserTweets", **BASE_CONFIG )) xhr_calls = result.scrape_result["browser_data"]["xhr_call"] for xhr in xhr_calls: if "UserTweets" not in xhr["url"] or not xhr.get("response"): continue data = json.loads(xhr["response"]["body"]) instructions = data["data"]["user"]["result"]["timeline"]["timeline"]["instructions"] for instruction in instructions: for entry in instruction.get("entries", []): item = entry.get("content", {}).get("itemContent", {}) if item.get("__typename") != "TimelineTweet": continue user = item["tweet_results"]["result"]["core"]["user_results"]["result"] if user.get("rest_id"): return parse_profile(user) ``` **Step 3: Flatten the fields you care about.** The raw object nests most data under `legacy`, so pull the useful keys to the top. python```python def parse_profile(data: dict) -> dict: return { "id": data["id"], "rest_id": data["rest_id"], "verified": data["is_blue_verified"], **data["legacy"], } ``` Running `scrape_profile("https://x.com/x")` returns a flat profile record: Example Outputjson```json { "id": "VXNlcjo3ODMyMTQ=", "rest_id": "783214", "verified": true, "description": "what's happening?!", "followers_count": 67000000, "friends_count": 0, "listed_count": 88000, "media_count": 2200, "statuses_count": 14000, "profile_banner_url": "https://pbs.twimg.com/profile_banners/783214/1690175171", "url": "https://t.co/bgVDp1Y6L8" } ``` You get the display name, bio, follower and following counts, verification status, and media counts. For the same hidden-API pattern on a sibling platform, see the [How To Scrape TikTok in 2026](https://scrapfly.io/blog/posts/how-to-scrape-tiktok-python-json). Tweets work almost identically. ## How to Scrape Tweets (X Posts) with Python You scrape an X post by loading its status URL and capturing the `TweetResultByRestId` GraphQL response. The same browser approach applies, with a different selector and filter. **Step 1: Load the tweet and grab the response.** Wait for the tweet element, then filter the background calls. python```python async def scrape_tweet(url: str) -> dict: result = await SCRAPFLY.async_scrape(ScrapeConfig( url, auto_scroll=True, wait_for_selector="[data-testid='tweet']", **BASE_CONFIG )) xhr_calls = result.scrape_result["browser_data"]["xhr_call"] for xhr in xhr_calls: if "TweetResultByRestId" not in xhr["url"] or not xhr.get("response"): continue data = json.loads(xhr["response"]["body"]) return parse_tweet(data["data"]["tweetResult"]["result"]) ``` **Step 2: Parse the fields with jmespath.** The tweet data nests under `legacy`, so a single query flattens the important parts. python```python import jmespath def parse_tweet(data: dict) -> dict: return jmespath.search("""{ created_at: legacy.created_at, text: legacy.full_text, favorite_count: legacy.favorite_count, retweet_count: legacy.retweet_count, reply_count: legacy.reply_count, quote_count: legacy.quote_count, bookmark_count: legacy.bookmark_count, views: views.count, id: legacy.id_str, conversation_id: legacy.conversation_id_str, language: legacy.lang }""", data) ``` Calling `scrape_tweet` on a status URL returns the post and its engagement counts: Example Outputjson```json { "created_at": "Thu Jun 01 13:47:03 +0000 2023", "text": "A new blog post has been published! \n\nTop 10 Web Scraping Packages for Python 🤖", "favorite_count": 8, "retweet_count": 1, "reply_count": 7, "quote_count": 0, "bookmark_count": 1, "views": "2296", "id": "1664267318053179398", "conversation_id": "1664267318053179398", "language": "en" } ``` You get the text, timestamp, engagement counts, and view count for any public tweet. Note that replies and search aren't reachable this way, since both need a login. Before you can scrape a tweet, though, you need its URL. ## How to Find Tweet and Profile URLs to Scrape X.com's own search is login-walled, but Google indexes public tweets. A `site:` query is often the fastest way to find scrape targets without an account. Use these patterns in Google directly: - `site:x.com inurl:status ` finds individual tweets about a topic. - `site:x.com ` finds profile pages for a person or brand. - Add Google's date-range tools to filter for recent posts. One honest caveat: Google's index lags real-time Twitter by hours to days. For breaking content, it won't be current, but for building a target list of profiles and posts, it's the practical workaround. Once you have URLs, pacing decides whether your scraper survives. ## Rate Limits, Proxies, and Pacing X.com rate-limits by IP and session, so a working Twitter scraper is as much about pacing as parsing. A single datacenter IP gets blocked within a request or two, and anonymous views cap out around 300 requests per hour. Follow these operational rules to stay under the radar: - **Pace your requests.** Leave a short delay between calls instead of firing them back to back. - **Cap concurrency.** Keep parallel workers low; a handful at a time, not dozens. - **Back off on 429s.** When X.com returns a rate-limit response, wait and retry with increasing delay. - **Cache and deduplicate.** Never fetch the same tweet or profile twice in one run. - **Rotate residential IPs.** Datacenter ranges are pre-flagged, so route through residential proxies. This kind of crisp pacing is exactly what tutorials like [AIMultiple](https://aimultiple.com/twitter-web-scraping) (May 2026) get cited for. Residential proxies matter most because requests come from real home IP addresses, which X.com's detection treats as ordinary users. For a sense of scale, scraping 10,000 tweets runs roughly 50-100 requests and transfers 5-10 MB. That costs a few dollars at standard residential proxy rates. For deeper background, see the [The Complete Guide To Using Proxies For Web Scraping](https://scrapfly.io/blog/posts/introduction-to-proxies-in-web-scraping) and [How to Rotate Proxies in Web Scraping](https://scrapfly.io/blog/posts/how-to-rotate-proxies-in-web-scraping). Pacing and proxies are the difference between a scraper that runs once and one that runs daily. ## Scrape Twitter with Scrapfly Scrapfly provides web scraping, screenshot, and extraction APIs for data collection at scale. For X.com, it handles the guest tokens, residential proxy rotation, and browser rendering that break DIY scrapers at production volume. - [Anti-Scraping Protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - automatically defeats Cloudflare, DataDome, PerimeterX, Akamai, and 90+ other bot systems with `asp=True`. - [Smart proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) - residential and datacenter pools with country and ASN level geo-targeting. - [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - render X.com's GraphQL responses through real cloud browsers. - [Session management](https://scrapfly.io/docs/scrape-api/session) - keep cookies, headers, and IPs consistent so guest tokens stay valid. - [Python SDK](https://scrapfly.io/docs/sdk/python) with async support for parallel profile and tweet scraping. ### Web Scraping API Scrape any website with our powerful API. Anti-bot bypass, JavaScript rendering, and rotating proxies built-in. [Try Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started) ## FAQ Can I scrape Twitter without an API key?Yes. Public profiles and tweets are reachable without the official API by rendering the page and capturing its GraphQL responses, as the code above does. Protected accounts and search still need a login. What replaced snscrape for scraping Twitter?Nothing free and stable. The working paths are session-based libraries that carry account-ban risk, DIY browser automation, or a maintained scraper API. How often does X.com change its scraping defenses?Every 2-4 weeks, X.com adjusts guest tokens, doc\_ids, rate limits, or detection patterns. There's no fixed schedule, so changes land whenever they land. Why do datacenter proxies fail on Twitter?X.com detects datacenter IP ranges and blocks them within a request or two. Even with valid tokens and doc\_ids, a datacenter proxy fails almost immediately, so residential IPs are required. Can you scrape Twitter followers?No. Full follower and following lists are login-gated, so no anonymous tool can reach them. You can still read the public follower count from a profile. ## Summary Public Twitter data is still scrapeable in 2026, but the easy paths are gone. The official API is priced for enterprises, and the free libraries snscrape, Twint, and Nitter are dead. DIY browser automation works but eats 10-15 hours of maintenance a month. A maintained scraper API trades cost for uptime. The reason DIY scrapers break isn't the HTML. It's the hidden GraphQL API and its three rotating guards: guest tokens, doc\_ids, and rate limits. A browser approach sidesteps doc\_id tracking, and residential proxies with careful pacing keep you unblocked. Pick by volume, uptime tolerance, and budget. For production pipelines that can't afford weekly breakage, the open-source [twitter-scraper](https://github.com/scrapfly/scrapfly-scrapers/tree/main/twitter-scraper) and the [Web Scraping API](https://scrapfly.io/web-scraping-api) behind it survive X.com's next change. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [Can You Still Scrape Twitter in 2026?](#can-you-still-scrape-twitter-in-2026) - [What Twitter Data Can You Scrape (and What's Off-Limits)?](#what-twitter-data-can-you-scrape-and-what-s-off-limits) - [Is It Legal to Scrape Twitter (X)?](#is-it-legal-to-scrape-twitter-x) - [4 Ways to Get Twitter Data in 2026](#4-ways-to-get-twitter-data-in-2026) - [Option 1: The Official X API (Priced Out for Most)](#option-1-the-official-x-api-priced-out-for-most) - [Option 2: Free Scraper Libraries: snscrape, Twint, and Nitter (RIP)](#option-2-free-scraper-libraries-snscrape-twint-and-nitter-rip) - [Option 3: Build Your Own with Playwright or Selenium](#option-3-build-your-own-with-playwright-or-selenium) - [Option 4: A Maintained Twitter Scraper API](#option-4-a-maintained-twitter-scraper-api) - [How Does Twitter's Hidden API Work?](#how-does-twitter-s-hidden-api-work) - [Guest Tokens: Twitter's Session Gatekeeper](#guest-tokens-twitter-s-session-gatekeeper) - [doc\_ids: Why Scrapers Break Every Few Weeks](#doc-ids-why-scrapers-break-every-few-weeks) - [What Changed Between 2023 and 2026](#what-changed-between-2023-and-2026) - [Project Setup](#project-setup) - [How to Scrape Twitter Profiles with Python](#how-to-scrape-twitter-profiles-with-python) - [How to Scrape Tweets (X Posts) with Python](#how-to-scrape-tweets-x-posts-with-python) - [How to Find Tweet and Profile URLs to Scrape](#how-to-find-tweet-and-profile-urls-to-scrape) - [Rate Limits, Proxies, and Pacing](#rate-limits-proxies-and-pacing) - [Scrape Twitter with Scrapfly](#scrape-twitter-with-scrapfly) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Explore this Article with AI [ ChatGPT ](https://chat.openai.com/?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-twitter) [ Gemini ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-twitter) [ Grok ](https://x.com/i/grok?text=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-twitter) [ Perplexity ](https://www.perplexity.ai/search/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-twitter) [ Claude ](https://claude.ai/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-twitter) ## Related Articles [ http python ### Web Scraping with Python Introduction tutorial to web scraping with Python. How to collect and parse public data. Challenges, best practices and ... ](https://scrapfly.io/blog/posts/web-scraping-with-python) [ python scrapeguide ### How to Scrape Reddit Posts, Subreddits and Profiles In this article, we'll explore how to scrape Reddit. We'll extract various social data types from subreddits, posts, and... ](https://scrapfly.io/blog/posts/how-to-scrape-reddit-social-data) [ python data-parsing ### How to Scrape Government and Public Records Data Map the public-data sources worth scraping, the friction each portal throws at you, and how to get past it and normalize... ](https://scrapfly.io/blog/posts/scraping-government-public-records-data) Scale your web scraping effortlessly, **1,000 free credits** [Start Free](https://scrapfly.io/register)