How to Scrape Twitter with Python (2023-07 Update)

article feature image

With the recent news of Twitter closing its API to new developers, we decided to write a tutorial on how to scrape Twitter.

In this tutorial, we'll be using nothing but Python to retrieve Twitter data such as:

  • Tweet information and replies.
  • Twitter user profile information.
  • Discover Tweets through Twitter topic timelines

We'll be scraping Twitter without login or any complex tricks using headless browsers and capturing background requests making this a very simple and powerful scraper.

For our headless browser environment, we'll be using Scrapfly SDK with Javascript Rendering feature. Alternatively, for non scraply users we'll also show how to achieve similar results using Playwright.

Latest twitter.com Scraper Code

https://github.com/scrapfly/scrapfly-scrapers/

Why Scrape Twitter?

Twitter is a major announcement hub where people and companies publish their announcements. This is a great opportunity to use Twitter to follow industry trends. For example, stock market or crypto market targets could be scraped to predict the future price of a stock or crypto.

Twitter is also a great source of data for sentiment analysis. You can use Twitter to find out what people think about a certain topic or brand. This is useful for market research, product development, and brand awareness.

So, if we can scrape Twitter data with Python we can have access to this valuable public information for free!

Project Setup

In this tutorial, we'll cover Twitter scraping using Python and Scrapfly-sdk or Playwright.

To parse the scraped Twitter datasets we'll be using Jmespath JSON parsing library which allows to reshape JSON data and nested-lookup which allows to search nested JSON data.

All of these libraries are available for free and can be installed via pip install terminal command:

$ pip install playwright jmespath nested-lookup "scrapfly-sdk[all]" 
Web Scraping with Playwright and Python

For an introduction to web scraping with Playwright see this beginner's guide which covers common functionality and an example project.

Web Scraping with Playwright and Python

How Does Twitter Work?

Before we start scraping, let's take a quick look at how does the Twitter.com website works so we can understand how to scrape it.

To start, Twitter is a javascript web application so scraping it without a headless browser such as Playwright or Scrapfly SDK would be very difficult as we'd have to reverse engineer the entire Twitter API and application process.

Next, Twitter page HTML is dynamic and complex making parsing scraped content very difficult.

So, the best approach to scrape Twitter is to use a headless browser and capture background requests that download the Tweet and user data.

For example, if we take a look at a Twitter profile page in Browser Developer Tools we can see the requests Twitter performs in the background to load the page data:

0:00
/
We can see Twitter backend making a request in the background to retrieve the data

So, to scrape Twitter we'll be using a headless browser (like Playwright) and capture these background requests to retrieve the data we need!

2023-07-02: Bypass Twitter Login Requirement for Posts and Timelines

Recently, Twitter has introduced a temporary login requirement for viewing posts and pages. While this is a temporary measure, it does make scraping Twitter more difficult.

One way to get past this requirement is to scrape Twitter Tweet and Timeline embed widget instead of the website itself.

Let's take a look at how to scrape Twitter embed using Python.

Timeline Bypass

To scrape Twitter Timelines we can use the timeline embed widget. This widget can be found via URL pattern:

https://syndication.twitter.com/srv/timeline-profile/screen-name/USER
# e.g. 
https://syndication.twitter.com/srv/timeline-profile/screen-name/scrapfly_dev

To scrape it using python

Python
ScrapFly
import json
import httpx
from parsel import Selector

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.54 Safari/537.36",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.9",
}


# retrieve embed HTML
with httpx.Client(http2=True, headers=HEADERS) as client:
    response = client.get(
        "https://syndication.twitter.com/srv/timeline-profile/screen-name/scrapfly_dev"
    )
    assert response.status_code == 200
sel = Selector(response.text)
# find data cache:
data = json.loads(sel.css("script#__NEXT_DATA__::text").get())
# parse tweet data from data cache JSON:
tweet_data = data["props"]["pageProps"]["timeline"]["entries"]
tweets = [tweet["content"]["tweet"] for tweet in tweet_data]
print(tweets)
import json
from scrapfly import ScrapflyClient, ScrapeConfig

scrapfly = ScrapflyClient("YOUR SCRAPFLY KEY")

result = scrapfly.scrape(
    ScrapeConfig("https://syndication.twitter.com/srv/timeline-profile/screen-name/scrapfly_dev")
)
# find data cache:
data = json.loads(result.selector.css("script#__NEXT_DATA__::text").get())
# parse tweet data from data cache JSON:
tweet_data = data["props"]["pageProps"]["timeline"]["entries"]
tweets = [tweet["content"]["tweet"] for tweet in tweet_data]
print(tweets)

In the example above, we're scraping the Twitter-embedded timeline and then parsing hidden web data for all timeline tweet information. For refining the dataset see the Tweet Parsing section.

Tweet Bypass

To scrape Tweets without login we can use the Tweet embed widget. This widget is can be accessed through url pattern:

https://platform.twitter.com/embed/Tweet.html?id=TWEET_ID
# where TWEET_ID is numeric id of the tweet
https://twitter.com/Scrapfly_dev/status/1664267318053179398 
#                                       ^^^^^^^^^^^^^^^^^^
https://platform.twitter.com/embed/Tweet.html?id=1664267318053179398

However, that's not the direct URL to the Tweet's data. If we take a look at what the browser does when we visit this page:

0:00
/
Background request is being made once Tweet Embed is viewed

So, to scrape this we'll be scraping this URL instead:

https://cdn.syndication.twimg.com/tweet-result?id=1664267318053179398&lang=en

Which is very simply done using Python:

Python
ScrapFly
import httpx

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.54 Safari/537.36",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.9",
}


# retrieve embed HTML
with httpx.Client(http2=True, headers=HEADERS) as client:
    response = client.get(
        "https://cdn.syndication.twimg.com/tweet-result?id=1664267318053179398&lang=en",
    )
    assert response.status_code == 200
    data = response.json()
    print(data)

import json
from scrapfly import ScrapflyClient, ScrapeConfig

scrapfly = ScrapflyClient("YOUR SCRAPFLY KEY")

result = scrapfly.scrape(
    ScrapeConfig("https://cdn.syndication.twimg.com/tweet-result?id=1664267318053179398&lang=en")
)
data = json.loads(result.content)
print(data)

Above we're calling embed tweet widget API directly though Twitter's CDN. By providing Tweet numeric ID we can scrape the entire dataset of the Tweet.

Scraping Tweets

To scrape individual tweet pages we'll be loading the page using a headless browser and capturing the background requests that connect to TweetDetail graphql endpoint.

This background requests returns a JSON response that contains all of the Tweet and user information.

So, to scrape this using Python we can either use Playwright or Scrapfly SDK:

Python
ScrapFly
from playwright.sync_api import sync_playwright
from nested_lookup import nested_lookup


def scrape_tweet(url: str) -> dict:
    """
    Scrape a single tweet page for Tweet thread e.g.:
    https://twitter.com/Scrapfly_dev/status/1667013143904567296
    Return parent tweet, reply tweets and recommended tweets
    """
    _xhr_calls = []

    def intercept_response(response):
        """capture all background requests and save them"""
        # we can extract details from background requests
        if response.request.resource_type == "xhr":
            _xhr_calls.append(response)
        return response

    with sync_playwright() as pw:
        browser = pw.chromium.launch()
        context = browser.new_context(viewport={"width": 1920, "height": 1080})
        page = context.new_page()

        # enable background request intercepting:
        page.on("response", intercept_response)
        # go to url and wait for the page to load
        page.goto(url)
        page.wait_for_selector("[data-testid='tweet']")

        # find all tweet background requests:
        tweet_calls = [f for f in _xhr_calls if "TweetDetail" in f.url]
        tweets = []
        for xhr in tweet_calls:
            data = xhr.json()
            xhr_tweets = nested_lookup("tweet_results", data)
            tweets.extend([tweet["result"] for tweet in xhr_tweets])

        # Now that we have all tweets we can parse them into a thread
        # The first tweet is the parent, the rest are replies or suggested tweets
        parent = tweets.pop(0)
        replies = []
        other = []
        for tweet in tweets:
            if tweet["conversation_id"] == parent["conversation_id"]:
                replies.append(tweet)
            else:
                other.append(tweet)
        return {
            "tweet": parent,
            "replies": replies,
            "other": other,  # ads, recommended etc
        }


if __name__ == "__main__":
    print(scrape_tweet("https://twitter.com/Scrapfly_dev/status/1664267318053179398"))
import json
from typing import Dict

from nested_lookup import nested_lookup
from scrapfly import ScrapeConfig, ScrapflyClient

SCRAPFLY = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"])
BASE_CONFIG = {
    # Twitter.com requires Anti Scraping Protection bypass feature.
    # for more: https://scrapfly.io/docs/scrape-api/anti-scraping-protection
    "asp": True,
    # Twitter.com is javascript-powered web application so it requires
    # headless browsers for scraping
    "render_js": True,
    "country": "CA",  # set prefered country here, for example Canada
}

async def scrape_tweet(url: str) -> Dict:
    """
    Scrape a single tweet page for Tweet thread e.g.:
    https://twitter.com/Scrapfly_dev/status/1667013143904567296
    Return parent tweet, reply tweets and recommended tweets
    """
    result = await scrapfly.async_scrape(ScrapeConfig(
        url, 
        wait_for_selector="[data-testid='tweet']"
        **BASE_CONFIG
    ))
    # capture background requests and extract ones that request Tweet data
    _xhr_calls = result.scrape_result["browser_data"]["xhr_call"]
    tweet_call = [f for f in _xhr_calls if "TweetDetail" in f["url"]]
    tweets = []
    for xhr in tweet_call:
        if not xhr["response"]:
            continue
        data = json.loads(xhr["response"]["body"])
        # find tweet_results key recursive as that's where tweet data is located
        xhr_tweets = nested_lookup("tweet_results", data)
        tweets.extend([parse_tweet(tweet["result"]) for tweet in xhr_tweets])

    # Now that we have all tweets we can parse them into a thread
    # The first tweet is the parent, the rest are replies or suggested tweets
    parent = tweets.pop(0)
    replies = []
    other = []
    for tweet in tweets:
        if tweet["conversation_id"] == parent["conversation_id"]:
            replies.append(tweet)
        else:
            other.append(tweet)
    return {
        "tweet": parent,
        "replies": replies,
        "other": other,  # ads, recommended etc
    }
  
if __name__ == "__main__":
    import asyncio
    asyncio.run(scrape_tweet("https://twitter.com/Scrapfly_dev/status/1664267318053179398")

Example Output
{
  "tweet": {
    "__typename": "Tweet",
    "rest_id": "1664267318053179398",
    "core": {
      "user_results": {
        "result": {
          "__typename": "User",
          "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
          "rest_id": "1310623081300402178",
          "affiliates_highlighted_label": {},
          "is_blue_verified": true,
          "profile_image_shape": "Circle",
          "legacy": {
            "created_at": "Mon Sep 28 16:51:22 +0000 2020",
            "default_profile": true,
            "default_profile_image": false,
            "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
            "entities": {
              "description": {
                "urls": []
              },
              "url": {
                "urls": [
                  {
                    "display_url": "scrapfly.io",
                    "expanded_url": "https://scrapfly.io",
                    "url": "https://t.co/1Is3k6KzyM",
                    "indices": [
                      0,
                      23
                    ]
                  }
                ]
              }
            },
            "fast_followers_count": 0,
            "favourites_count": 26,
            "followers_count": 163,
            "friends_count": 993,
            "has_custom_timelines": true,
            "is_translator": false,
            "listed_count": 2,
            "location": "Paris",
            "media_count": 11,
            "name": "Scrapfly",
            "normal_followers_count": 163,
            "pinned_tweet_ids_str": [],
            "possibly_sensitive": false,
            "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
            "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
            "profile_interstitial_type": "",
            "screen_name": "Scrapfly_dev",
            "statuses_count": 56,
            "translator_type": "none",
            "url": "https://t.co/1Is3k6KzyM",
            "verified": false,
            "withheld_in_countries": []
          }
        }
      }
    },
    "edit_control": {
      "edit_tweet_ids": [
        "1664267318053179398"
      ],
      "editable_until_msecs": "1685629023000",
      "is_edit_eligible": true,
      "edits_remaining": "5"
    },
    "is_translatable": false,
    "views": {
      "count": "43",
      "state": "EnabledWithCount"
    },
    "source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
    "legacy": {
      "bookmark_count": 0,
      "bookmarked": false,
      "created_at": "Thu Jun 01 13:47:03 +0000 2023",
      "conversation_id_str": "1664267318053179398",
      "display_text_range": [
        0,
        122
      ],
      "entities": {
        "media": [
          {
            "display_url": "pic.twitter.com/zLjDlxdKee",
            "expanded_url": "https://twitter.com/Scrapfly_dev/status/1664267318053179398/photo/1",
            "id_str": "1664267314160607232",
            "indices": [
              123,
              146
            ],
            "media_url_https": "https://pbs.twimg.com/media/FxiqTffWIAALf7O.png",
            "type": "photo",
            "url": "https://t.co/zLjDlxdKee",
            "features": {
              "large": {
                "faces": []
              },
              "medium": {
                "faces": []
              },
              "small": {
                "faces": []
              },
              "orig": {
                "faces": []
              }
            },
            "sizes": {
              "large": {
                "h": 416,
                "w": 796,
                "resize": "fit"
              },
              "medium": {
                "h": 416,
                "w": 796,
                "resize": "fit"
              },
              "small": {
                "h": 355,
                "w": 680,
                "resize": "fit"
              },
              "thumb": {
                "h": 150,
                "w": 150,
                "resize": "crop"
              }
            },
            "original_info": {
              "height": 416,
              "width": 796,
              "focus_rects": [
                {
                  "x": 27,
                  "y": 0,
                  "w": 743,
                  "h": 416
                },
                {
                  "x": 190,
                  "y": 0,
                  "w": 416,
                  "h": 416
                },
                {
                  "x": 216,
                  "y": 0,
                  "w": 365,
                  "h": 416
                },
                {
                  "x": 294,
                  "y": 0,
                  "w": 208,
                  "h": 416
                },
                {
                  "x": 0,
                  "y": 0,
                  "w": 796,
                  "h": 416
                }
              ]
            }
          }
        ],
        "user_mentions": [],
        "urls": [
          {
            "display_url": "scrapfly.io/blog/top-10-we\u2026",
            "expanded_url": "https://scrapfly.io/blog/top-10-web-scraping-libraries-in-python/",
            "url": "https://t.co/d2iFdAV2LJ",
            "indices": [
              99,
              122
            ]
          }
        ],
        "hashtags": [],
        "symbols": []
      },
      "extended_entities": {
        "media": [
          {
            "display_url": "pic.twitter.com/zLjDlxdKee",
            "expanded_url": "https://twitter.com/Scrapfly_dev/status/1664267318053179398/photo/1",
            "id_str": "1664267314160607232",
            "indices": [
              123,
              146
            ],
            "media_key": "3_1664267314160607232",
            "media_url_https": "https://pbs.twimg.com/media/FxiqTffWIAALf7O.png",
            "type": "photo",
            "url": "https://t.co/zLjDlxdKee",
            "ext_media_availability": {
              "status": "Available"
            },
            "features": {
              "large": {
                "faces": []
              },
              "medium": {
                "faces": []
              },
              "small": {
                "faces": []
              },
              "orig": {
                "faces": []
              }
            },
            "sizes": {
              "large": {
                "h": 416,
                "w": 796,
                "resize": "fit"
              },
              "medium": {
                "h": 416,
                "w": 796,
                "resize": "fit"
              },
              "small": {
                "h": 355,
                "w": 680,
                "resize": "fit"
              },
              "thumb": {
                "h": 150,
                "w": 150,
                "resize": "crop"
              }
            },
            "original_info": {
              "height": 416,
              "width": 796,
              "focus_rects": [
                {
                  "x": 27,
                  "y": 0,
                  "w": 743,
                  "h": 416
                },
                {
                  "x": 190,
                  "y": 0,
                  "w": 416,
                  "h": 416
                },
                {
                  "x": 216,
                  "y": 0,
                  "w": 365,
                  "h": 416
                },
                {
                  "x": 294,
                  "y": 0,
                  "w": 208,
                  "h": 416
                },
                {
                  "x": 0,
                  "y": 0,
                  "w": 796,
                  "h": 416
                }
              ]
            }
          }
        ]
      },
      "favorite_count": 0,
      "favorited": false,
      "full_text": "A new blog post has been published! \n\nTop 10 Web Scraping Packages for Python \ud83e\udd16\n\nCheckout it out \ud83d\udc47\nhttps://t.co/d2iFdAV2LJ https://t.co/zLjDlxdKee",
      "is_quote_status": false,
      "lang": "en",
      "possibly_sensitive": false,
      "possibly_sensitive_editable": true,
      "quote_count": 0,
      "reply_count": 0,
      "retweet_count": 0,
      "retweeted": false,
      "user_id_str": "1310623081300402178",
      "id_str": "1664267318053179398"
    },
    "quick_promote_eligibility": {
      "eligibility": "IneligibleUserUnauthorized"
    }
  },
  "replies": [],
  "other": []
}

Here, we loaded the Tweet page using a headless browser and captured all of the background requests. Then, we filtered out the ones that contained the Tweet data and extract that.

One important note here is that we need to wait for the page to load which is indicated by tweets appearing on the page HTML otherwise we'll return our scrape before the background requests have finished.

This resulted in a massive JSON dataset that can be difficult to work with. So, let's take a look at how to reduce it with a bit of JSON parsing next.

Parsing Tweet Dataset

The Tweet dataset we scraped contains a lot of complex data so let's reduce it to something more clean and simple using Jmespath JSON parsing library.

For this, we'll be using jmespath's JSON reshaping feature which allows us to rename keys and flatten nested objects:

from typing import Dict

def parse_tweet(data: Dict) -> Dict:
    """Parse Twitter tweet JSON dataset for the most important fields"""
    result = jmespath.search(
        """{
        created_at: legacy.created_at,
        attached_urls: legacy.entities.urls[].expanded_url,
        attached_urls2: legacy.entities.url.urls[].expanded_url,
        attached_media: legacy.entities.media[].media_url_https,
        tagged_users: legacy.entities.user_mentions[].screen_name,
        tagged_hashtags: legacy.entities.hashtags[].text,
        favorite_count: legacy.favorite_count,
        bookmark_count: legacy.bookmark_count,
        quote_count: legacy.quote_count,
        reply_count: legacy.reply_count,
        retweet_count: legacy.retweet_count,
        quote_count: legacy.quote_count,
        text: legacy.full_text,
        is_quote: legacy.is_quote_status,
        is_retweet: legacy.retweeted,
        language: legacy.lang,
        user_id: legacy.user_id_str,
        id: legacy.id_str,
        conversation_id: legacy.conversation_id_str,
        source: source,
        views: views.count
    }""",
        data,
    )
    result["poll"] = {}
    poll_data = jmespath.search("card.legacy.binding_values", data) or []
    for poll_entry in poll_data:
        key, value = poll_entry["key"], poll_entry["value"]
        if "choice" in key:
            result["poll"][key] = value["string_value"]
        elif "end_datetime" in key:
            result["poll"]["end"] = value["string_value"]
        elif "last_updated_datetime" in key:
            result["poll"]["updated"] = value["string_value"]
        elif "counts_are_final" in key:
            result["poll"]["ended"] = value["boolean_value"]
        elif "duration_minutes" in key:
            result["poll"]["duration"] = value["string_value"]
    user_data = jmespath.search("core.user_results.result", data)
    if user_data:
        result["user"] = parse_user(user_data)
    return result

Above we're using jmespath to reshape the giant, nested dataset we scraped from Twitter's graphql backend into a flat dictionary containing only the most important fields.

Scraping Twitter User Profiles

To scrape Twitter profile pages we'll be using the same background request capturing approach just this time we'll be capturing UserBy endpoints which contain the user data in addition to UserTweets endpoint which contains the user's tweets.

We'll be using the same technique we used to scrape Tweets just with a few minor adjustments:

Python
ScrapFly
from playwright.sync_api import sync_playwright
from nested_lookup import nested_lookup


def scrape_profile(url: str) -> dict:
    """
    Scrapes Twitter user profile page e.g.:
    https://twitter.com/scrapfly_dev
    returns user data and latest tweets
    """
    _xhr_calls = []

    def intercept_response(response):
        """capture all background requests and save them"""
        # we can extract details from background requests
        if response.request.resource_type == "xhr":
            _xhr_calls.append(response)
        return response

    with sync_playwright() as pw:
        browser = pw.chromium.launch()
        context = browser.new_context(viewport={"width": 1920, "height": 1080})
        page = context.new_page()
        # enable intercepting for this page

        page.on("response", intercept_response)
        page.goto(url)
        page.wait_for_selector("[data-testid='tweet']")

        user_calls = [f for f in _xhr_calls if "UserBy" in f.url]
        users = {}
        for xhr in user_calls:
            data = xhr.json()
            user_data = data["data"]["user"]["result"]
            users[user_data["legacy"]["screen_name"]] = user_data

        tweet_calls = [f for f in _xhr_calls if "UserTweets" in f.url]
        tweets = []
        for xhr in tweet_calls:
            data = xhr.json()
            xhr_tweets = nested_lookup("tweet_results", data)
            tweets.extend([tweet["result"] for tweet in xhr_tweets])
            users[user_data["legacy"]["screen_name"]] = user_data

    return {"users": users, "tweets": tweets}


if __name__ == "__main__":
    print(scrape_profile("https://twitter.com/Scrapfly_dev"))

import json
from typing import Dict

from nested_lookup import nested_lookup
from scrapfly import ScrapeConfig, ScrapflyClient

SCRAPFLY = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"])
BASE_CONFIG = {
    # Twitter.com requires Anti Scraping Protection bypass feature.
    # for more: https://scrapfly.io/docs/scrape-api/anti-scraping-protection
    "asp": True,
    # Twitter.com is javascript-powered web application so it requires
    # headless browsers for scraping
    "render_js": True,
    "country": "CA",  # set prefered country here, for example Canada
}


async def scrape_profile(url: str) -> Dict:
    """
    Scrapes Twitter user profile page e.g.:
    https://twitter.com/scrapfly_dev
    returns user data and latest tweets
    """
    result = await scrapfly.async_scrape(ScrapeConfig(
        url, 
        wait_for_selector="[data-testid='tweet']"
        **BASE_CONFIG
    ))
    # capture background requests and extract ones that contain user data
    # and their latest tweets
    _xhr_calls = result.scrape_result["browser_data"]["xhr_call"]
    user_calls = [f for f in _xhr_calls if "UserBy" in f["url"]]
    users = {}
    for xhr in user_calls:
        data = json.loads(xhr["response"]["body"])
        parsed = parse_user(data["data"]["user"]["result"])
        users[parsed["screen_name"]] = parsed

    tweet_paging_calls = [f for f in _xhr_calls if "UserTweets" in f["url"]]
    tweets = []
    for xhr in tweet_paging_calls:
        data = json.loads(xhr["response"]["body"])
        xhr_tweets = nested_lookup("tweet_results", data)
        tweets.extend([parse_tweet(tweet["result"]) for tweet in xhr_tweets])
    return {
        "tweets": tweets,
        "users": users,
    }

if __name__ == "__main__":
    import asyncio
    asyncio.run(scrape_profile("https://twitter.com/Scrapfly_dev")
Example Output
{
  "users": {
    "Scrapfly_dev": {
      "__typename": "User",
      "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
      "rest_id": "1310623081300402178",
      "affiliates_highlighted_label": {},
      "is_blue_verified": true,
      "profile_image_shape": "Circle",
      "legacy": {
        "created_at": "Mon Sep 28 16:51:22 +0000 2020",
        "default_profile": true,
        "default_profile_image": false,
        "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
        "entities": {
          "description": {
            "urls": []
          },
          "url": {
            "urls": [
              {
                "display_url": "scrapfly.io",
                "expanded_url": "https://scrapfly.io",
                "url": "https://t.co/1Is3k6KzyM",
                "indices": [
                  0,
                  23
                ]
              }
            ]
          }
        },
        "fast_followers_count": 0,
        "favourites_count": 26,
        "followers_count": 163,
        "friends_count": 993,
        "has_custom_timelines": true,
        "is_translator": false,
        "listed_count": 2,
        "location": "Paris",
        "media_count": 11,
        "name": "Scrapfly",
        "normal_followers_count": 163,
        "pinned_tweet_ids_str": [],
        "possibly_sensitive": false,
        "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
        "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
        "profile_interstitial_type": "",
        "screen_name": "Scrapfly_dev",
        "statuses_count": 56,
        "translator_type": "none",
        "url": "https://t.co/1Is3k6KzyM",
        "verified": false,
        "withheld_in_countries": []
      },
      "business_account": {},
      "highlights_info": {
        "can_highlight_tweets": true,
        "highlighted_tweets": "0"
      },
      "creator_subscriptions_count": 0
    }
  },
  "tweets": [
    {
      "__typename": "Tweet",
      "rest_id": "1667013143904567296",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1667013143904567296"
        ],
        "editable_until_msecs": "1686285479000",
        "is_edit_eligible": true,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "35",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Fri Jun 09 03:37:59 +0000 2023",
        "conversation_id_str": "1667013143904567296",
        "display_text_range": [
          0,
          139
        ],
        "entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/mLnjGwMnmD",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1667013143904567296/photo/1",
              "id_str": "1667013141392179201",
              "indices": [
                140,
                163
              ],
              "media_url_https": "https://pbs.twimg.com/media/FyJrnrCWwAEpvzT.png",
              "type": "photo",
              "url": "https://t.co/mLnjGwMnmD",
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "medium": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 416,
                "width": 796,
                "focus_rects": [
                  {
                    "x": 27,
                    "y": 0,
                    "w": 743,
                    "h": 416
                  },
                  {
                    "x": 190,
                    "y": 0,
                    "w": 416,
                    "h": 416
                  },
                  {
                    "x": 216,
                    "y": 0,
                    "w": 365,
                    "h": 416
                  },
                  {
                    "x": 294,
                    "y": 0,
                    "w": 208,
                    "h": 416
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 796,
                    "h": 416
                  }
                ]
              }
            }
          ],
          "user_mentions": [],
          "urls": [
            {
              "display_url": "scrapfly.io/blog/parsing-d\u2026",
              "expanded_url": "https://scrapfly.io/blog/parsing-datetime-strings-with-python-and-dateparser/",
              "url": "https://t.co/FeEpxzQ3sK",
              "indices": [
                116,
                139
              ]
            }
          ],
          "hashtags": [],
          "symbols": []
        },
        "extended_entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/mLnjGwMnmD",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1667013143904567296/photo/1",
              "id_str": "1667013141392179201",
              "indices": [
                140,
                163
              ],
              "media_key": "3_1667013141392179201",
              "media_url_https": "https://pbs.twimg.com/media/FyJrnrCWwAEpvzT.png",
              "type": "photo",
              "url": "https://t.co/mLnjGwMnmD",
              "ext_media_availability": {
                "status": "Available"
              },
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "medium": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 416,
                "width": 796,
                "focus_rects": [
                  {
                    "x": 27,
                    "y": 0,
                    "w": 743,
                    "h": 416
                  },
                  {
                    "x": 190,
                    "y": 0,
                    "w": 416,
                    "h": 416
                  },
                  {
                    "x": 216,
                    "y": 0,
                    "w": 365,
                    "h": 416
                  },
                  {
                    "x": 294,
                    "y": 0,
                    "w": 208,
                    "h": 416
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 796,
                    "h": 416
                  }
                ]
              }
            }
          ]
        },
        "favorite_count": 0,
        "favorited": false,
        "full_text": "A new blog post has been published! \n\nHow to Parse Datetime Strings with Python and Dateparser \ud83e\udd16\n\nCheckout it out \ud83d\udc47\nhttps://t.co/FeEpxzQ3sK https://t.co/mLnjGwMnmD",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1667013143904567296"
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1664267318053179398",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1664267318053179398"
        ],
        "editable_until_msecs": "1685629023000",
        "is_edit_eligible": true,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "43",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Thu Jun 01 13:47:03 +0000 2023",
        "conversation_id_str": "1664267318053179398",
        "display_text_range": [
          0,
          122
        ],
        "entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/zLjDlxdKee",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1664267318053179398/photo/1",
              "id_str": "1664267314160607232",
              "indices": [
                123,
                146
              ],
              "media_url_https": "https://pbs.twimg.com/media/FxiqTffWIAALf7O.png",
              "type": "photo",
              "url": "https://t.co/zLjDlxdKee",
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "medium": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 416,
                "width": 796,
                "focus_rects": [
                  {
                    "x": 27,
                    "y": 0,
                    "w": 743,
                    "h": 416
                  },
                  {
                    "x": 190,
                    "y": 0,
                    "w": 416,
                    "h": 416
                  },
                  {
                    "x": 216,
                    "y": 0,
                    "w": 365,
                    "h": 416
                  },
                  {
                    "x": 294,
                    "y": 0,
                    "w": 208,
                    "h": 416
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 796,
                    "h": 416
                  }
                ]
              }
            }
          ],
          "user_mentions": [],
          "urls": [
            {
              "display_url": "scrapfly.io/blog/top-10-we\u2026",
              "expanded_url": "https://scrapfly.io/blog/top-10-web-scraping-libraries-in-python/",
              "url": "https://t.co/d2iFdAV2LJ",
              "indices": [
                99,
                122
              ]
            }
          ],
          "hashtags": [],
          "symbols": []
        },
        "extended_entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/zLjDlxdKee",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1664267318053179398/photo/1",
              "id_str": "1664267314160607232",
              "indices": [
                123,
                146
              ],
              "media_key": "3_1664267314160607232",
              "media_url_https": "https://pbs.twimg.com/media/FxiqTffWIAALf7O.png",
              "type": "photo",
              "url": "https://t.co/zLjDlxdKee",
              "ext_media_availability": {
                "status": "Available"
              },
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "medium": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 416,
                "width": 796,
                "focus_rects": [
                  {
                    "x": 27,
                    "y": 0,
                    "w": 743,
                    "h": 416
                  },
                  {
                    "x": 190,
                    "y": 0,
                    "w": 416,
                    "h": 416
                  },
                  {
                    "x": 216,
                    "y": 0,
                    "w": 365,
                    "h": 416
                  },
                  {
                    "x": 294,
                    "y": 0,
                    "w": 208,
                    "h": 416
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 796,
                    "h": 416
                  }
                ]
              }
            }
          ]
        },
        "favorite_count": 0,
        "favorited": false,
        "full_text": "A new blog post has been published! \n\nTop 10 Web Scraping Packages for Python \ud83e\udd16\n\nCheckout it out \ud83d\udc47\nhttps://t.co/d2iFdAV2LJ https://t.co/zLjDlxdKee",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1664267318053179398"
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1658170586508546048",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1658170586508546048"
        ],
        "editable_until_msecs": "1684175449000",
        "is_edit_eligible": true,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "45",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Mon May 15 18:00:49 +0000 2023",
        "conversation_id_str": "1658170586508546048",
        "display_text_range": [
          0,
          130
        ],
        "entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/1uRcNBdlmv",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1658170586508546048/photo/1",
              "id_str": "1658170581689196558",
              "indices": [
                131,
                154
              ],
              "media_url_https": "https://pbs.twimg.com/media/FwMBW9KWYA4B61a.png",
              "type": "photo",
              "url": "https://t.co/1uRcNBdlmv",
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 1070,
                  "w": 2048,
                  "resize": "fit"
                },
                "medium": {
                  "h": 627,
                  "w": 1200,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 1572,
                "width": 3008,
                "focus_rects": [
                  {
                    "x": 101,
                    "y": 0,
                    "w": 2807,
                    "h": 1572
                  },
                  {
                    "x": 718,
                    "y": 0,
                    "w": 1572,
                    "h": 1572
                  },
                  {
                    "x": 815,
                    "y": 0,
                    "w": 1379,
                    "h": 1572
                  },
                  {
                    "x": 1111,
                    "y": 0,
                    "w": 786,
                    "h": 1572
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 3008,
                    "h": 1572
                  }
                ]
              }
            }
          ],
          "user_mentions": [],
          "urls": [
            {
              "display_url": "scrapfly.io/blog/web-scrap\u2026",
              "expanded_url": "https://scrapfly.io/blog/web-scraping-microformats/",
              "url": "https://t.co/BhEcWS76rY",
              "indices": [
                107,
                130
              ]
            }
          ],
          "hashtags": [],
          "symbols": []
        },
        "extended_entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/1uRcNBdlmv",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1658170586508546048/photo/1",
              "id_str": "1658170581689196558",
              "indices": [
                131,
                154
              ],
              "media_key": "3_1658170581689196558",
              "media_url_https": "https://pbs.twimg.com/media/FwMBW9KWYA4B61a.png",
              "type": "photo",
              "url": "https://t.co/1uRcNBdlmv",
              "ext_media_availability": {
                "status": "Available"
              },
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 1070,
                  "w": 2048,
                  "resize": "fit"
                },
                "medium": {
                  "h": 627,
                  "w": 1200,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 1572,
                "width": 3008,
                "focus_rects": [
                  {
                    "x": 101,
                    "y": 0,
                    "w": 2807,
                    "h": 1572
                  },
                  {
                    "x": 718,
                    "y": 0,
                    "w": 1572,
                    "h": 1572
                  },
                  {
                    "x": 815,
                    "y": 0,
                    "w": 1379,
                    "h": 1572
                  },
                  {
                    "x": 1111,
                    "y": 0,
                    "w": 786,
                    "h": 1572
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 3008,
                    "h": 1572
                  }
                ]
              }
            }
          ]
        },
        "favorite_count": 0,
        "favorited": false,
        "full_text": "A new blog post has been published! \n\nWeb Scraping Simplified - Scraping Microformats \ud83e\udd16\n\nCheckout it out \ud83d\udc47\nhttps://t.co/BhEcWS76rY https://t.co/1uRcNBdlmv",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1658170586508546048"
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1653035176631447552",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1653035176631447552"
        ],
        "editable_until_msecs": "1682951072000",
        "is_edit_eligible": true,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "58",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Mon May 01 13:54:32 +0000 2023",
        "conversation_id_str": "1653035176631447552",
        "display_text_range": [
          0,
          122
        ],
        "entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/dlXAa03pFP",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1653035176631447552/photo/1",
              "id_str": "1653035174576246784",
              "indices": [
                123,
                146
              ],
              "media_url_https": "https://pbs.twimg.com/media/FvDCu63X0AAPJex.png",
              "type": "photo",
              "url": "https://t.co/dlXAa03pFP",
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "medium": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 416,
                "width": 796,
                "focus_rects": [
                  {
                    "x": 27,
                    "y": 0,
                    "w": 743,
                    "h": 416
                  },
                  {
                    "x": 190,
                    "y": 0,
                    "w": 416,
                    "h": 416
                  },
                  {
                    "x": 216,
                    "y": 0,
                    "w": 365,
                    "h": 416
                  },
                  {
                    "x": 294,
                    "y": 0,
                    "w": 208,
                    "h": 416
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 796,
                    "h": 416
                  }
                ]
              }
            }
          ],
          "user_mentions": [],
          "urls": [
            {
              "display_url": "scrapfly.io/blog/web-scrap\u2026",
              "expanded_url": "https://scrapfly.io/blog/web-scraping-with-python-httpx/",
              "url": "https://t.co/29fKBIVko3",
              "indices": [
                99,
                122
              ]
            }
          ],
          "hashtags": [],
          "symbols": []
        },
        "extended_entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/dlXAa03pFP",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1653035176631447552/photo/1",
              "id_str": "1653035174576246784",
              "indices": [
                123,
                146
              ],
              "media_key": "3_1653035174576246784",
              "media_url_https": "https://pbs.twimg.com/media/FvDCu63X0AAPJex.png",
              "type": "photo",
              "url": "https://t.co/dlXAa03pFP",
              "ext_media_availability": {
                "status": "Available"
              },
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "medium": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 416,
                "width": 796,
                "focus_rects": [
                  {
                    "x": 27,
                    "y": 0,
                    "w": 743,
                    "h": 416
                  },
                  {
                    "x": 190,
                    "y": 0,
                    "w": 416,
                    "h": 416
                  },
                  {
                    "x": 216,
                    "y": 0,
                    "w": 365,
                    "h": 416
                  },
                  {
                    "x": 294,
                    "y": 0,
                    "w": 208,
                    "h": 416
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 796,
                    "h": 416
                  }
                ]
              }
            }
          ]
        },
        "favorite_count": 1,
        "favorited": false,
        "full_text": "A new blog post has been published! \n\nHow to Web Scrape with HTTPX and Python \ud83e\udd16\n\nCheckout it out \ud83d\udc47\nhttps://t.co/29fKBIVko3 https://t.co/dlXAa03pFP",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1653035176631447552"
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1651042163885473794",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1651042163885473794"
        ],
        "editable_until_msecs": "1682475901000",
        "is_edit_eligible": true,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "55",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Wed Apr 26 01:55:01 +0000 2023",
        "conversation_id_str": "1651042163885473794",
        "display_text_range": [
          0,
          139
        ],
        "entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/lrWLxz3R4C",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1651042163885473794/photo/1",
              "id_str": "1651042161129840642",
              "indices": [
                140,
                163
              ],
              "media_url_https": "https://pbs.twimg.com/media/FumuGRNWYAIU0QI.png",
              "type": "photo",
              "url": "https://t.co/lrWLxz3R4C",
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 871,
                  "w": 1667,
                  "resize": "fit"
                },
                "medium": {
                  "h": 627,
                  "w": 1200,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 871,
                "width": 1667,
                "focus_rects": [
                  {
                    "x": 56,
                    "y": 0,
                    "w": 1555,
                    "h": 871
                  },
                  {
                    "x": 398,
                    "y": 0,
                    "w": 871,
                    "h": 871
                  },
                  {
                    "x": 451,
                    "y": 0,
                    "w": 764,
                    "h": 871
                  },
                  {
                    "x": 615,
                    "y": 0,
                    "w": 436,
                    "h": 871
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 1667,
                    "h": 871
                  }
                ]
              }
            }
          ],
          "user_mentions": [],
          "urls": [
            {
              "display_url": "scrapfly.io/blog/how-to-sc\u2026",
              "expanded_url": "https://scrapfly.io/blog/how-to-scrape-without-getting-blocked-tutorial/",
              "url": "https://t.co/tPqNzOy92c",
              "indices": [
                116,
                139
              ]
            }
          ],
          "hashtags": [],
          "symbols": []
        },
        "extended_entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/lrWLxz3R4C",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1651042163885473794/photo/1",
              "id_str": "1651042161129840642",
              "indices": [
                140,
                163
              ],
              "media_key": "3_1651042161129840642",
              "media_url_https": "https://pbs.twimg.com/media/FumuGRNWYAIU0QI.png",
              "type": "photo",
              "url": "https://t.co/lrWLxz3R4C",
              "ext_media_availability": {
                "status": "Available"
              },
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 871,
                  "w": 1667,
                  "resize": "fit"
                },
                "medium": {
                  "h": 627,
                  "w": 1200,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 871,
                "width": 1667,
                "focus_rects": [
                  {
                    "x": 56,
                    "y": 0,
                    "w": 1555,
                    "h": 871
                  },
                  {
                    "x": 398,
                    "y": 0,
                    "w": 871,
                    "h": 871
                  },
                  {
                    "x": 451,
                    "y": 0,
                    "w": 764,
                    "h": 871
                  },
                  {
                    "x": 615,
                    "y": 0,
                    "w": 436,
                    "h": 871
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 1667,
                    "h": 871
                  }
                ]
              }
            }
          ]
        },
        "favorite_count": 0,
        "favorited": false,
        "full_text": "A new blog post has been published! \n\nHow to Scrape Without Getting Blocked? In-Depth Tutorial \ud83e\udd16\n\nCheckout it out \ud83d\udc47\nhttps://t.co/tPqNzOy92c https://t.co/lrWLxz3R4C",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1651042163885473794"
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1647938361586221058",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1647938361586221058"
        ],
        "editable_until_msecs": "1681735897000",
        "is_edit_eligible": true,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "103",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Mon Apr 17 12:21:37 +0000 2023",
        "conversation_id_str": "1647938361586221058",
        "display_text_range": [
          0,
          106
        ],
        "entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/kCohTttrmM",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1647938361586221058/photo/1",
              "id_str": "1647938359170416642",
              "indices": [
                107,
                130
              ],
              "media_url_https": "https://pbs.twimg.com/media/Ft6nNN4XwAIuoHL.png",
              "type": "photo",
              "url": "https://t.co/kCohTttrmM",
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "medium": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 416,
                "width": 796,
                "focus_rects": [
                  {
                    "x": 27,
                    "y": 0,
                    "w": 743,
                    "h": 416
                  },
                  {
                    "x": 190,
                    "y": 0,
                    "w": 416,
                    "h": 416
                  },
                  {
                    "x": 216,
                    "y": 0,
                    "w": 365,
                    "h": 416
                  },
                  {
                    "x": 294,
                    "y": 0,
                    "w": 208,
                    "h": 416
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 796,
                    "h": 416
                  }
                ]
              }
            }
          ],
          "user_mentions": [],
          "urls": [],
          "hashtags": [],
          "symbols": []
        },
        "extended_entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/kCohTttrmM",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1647938361586221058/photo/1",
              "id_str": "1647938359170416642",
              "indices": [
                107,
                130
              ],
              "media_key": "3_1647938359170416642",
              "media_url_https": "https://pbs.twimg.com/media/Ft6nNN4XwAIuoHL.png",
              "type": "photo",
              "url": "https://t.co/kCohTttrmM",
              "ext_media_availability": {
                "status": "Available"
              },
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "medium": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 416,
                "width": 796,
                "focus_rects": [
                  {
                    "x": 27,
                    "y": 0,
                    "w": 743,
                    "h": 416
                  },
                  {
                    "x": 190,
                    "y": 0,
                    "w": 416,
                    "h": 416
                  },
                  {
                    "x": 216,
                    "y": 0,
                    "w": 365,
                    "h": 416
                  },
                  {
                    "x": 294,
                    "y": 0,
                    "w": 208,
                    "h": 416
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 796,
                    "h": 416
                  }
                ]
              }
            }
          ]
        },
        "favorite_count": 0,
        "favorited": false,
        "full_text": "A new blog post has been published! \n\nStepping into Footwear Market with Web Scraping \ud83e\udd16\n\nCheckout it out \ud83d\udc47 https://t.co/kCohTttrmM",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1647938361586221058"
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1646105015344418816",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1646105015344418816"
        ],
        "editable_until_msecs": "1681298793000",
        "is_edit_eligible": true,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "59",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Wed Apr 12 10:56:33 +0000 2023",
        "conversation_id_str": "1646105015344418816",
        "display_text_range": [
          0,
          131
        ],
        "entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/9WaCUH38NA",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1646105015344418816/photo/1",
              "id_str": "1646105012731363330",
              "indices": [
                132,
                155
              ],
              "media_url_https": "https://pbs.twimg.com/media/FtgjyaLXwAIrKBk.png",
              "type": "photo",
              "url": "https://t.co/9WaCUH38NA",
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "medium": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 416,
                "width": 796,
                "focus_rects": [
                  {
                    "x": 27,
                    "y": 0,
                    "w": 743,
                    "h": 416
                  },
                  {
                    "x": 190,
                    "y": 0,
                    "w": 416,
                    "h": 416
                  },
                  {
                    "x": 216,
                    "y": 0,
                    "w": 365,
                    "h": 416
                  },
                  {
                    "x": 294,
                    "y": 0,
                    "w": 208,
                    "h": 416
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 796,
                    "h": 416
                  }
                ]
              }
            }
          ],
          "user_mentions": [],
          "urls": [
            {
              "display_url": "Goat.com",
              "expanded_url": "http://Goat.com",
              "url": "https://t.co/PRRUEhuYJp",
              "indices": [
                52,
                75
              ]
            }
          ],
          "hashtags": [],
          "symbols": []
        },
        "extended_entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/9WaCUH38NA",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1646105015344418816/photo/1",
              "id_str": "1646105012731363330",
              "indices": [
                132,
                155
              ],
              "media_key": "3_1646105012731363330",
              "media_url_https": "https://pbs.twimg.com/media/FtgjyaLXwAIrKBk.png",
              "type": "photo",
              "url": "https://t.co/9WaCUH38NA",
              "ext_media_availability": {
                "status": "Available"
              },
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "medium": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 416,
                "width": 796,
                "focus_rects": [
                  {
                    "x": 27,
                    "y": 0,
                    "w": 743,
                    "h": 416
                  },
                  {
                    "x": 190,
                    "y": 0,
                    "w": 416,
                    "h": 416
                  },
                  {
                    "x": 216,
                    "y": 0,
                    "w": 365,
                    "h": 416
                  },
                  {
                    "x": 294,
                    "y": 0,
                    "w": 208,
                    "h": 416
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 796,
                    "h": 416
                  }
                ]
              }
            }
          ]
        },
        "favorite_count": 1,
        "favorited": false,
        "full_text": "A new blog post has been published! \n\nHow to Scrape https://t.co/PRRUEhuYJp for Fashion Apparel Data in Python \ud83e\udd16\n\nCheckout it out \ud83d\udc47 https://t.co/9WaCUH38NA",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1646105015344418816"
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1645521299136032769",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1645521299136032769"
        ],
        "editable_until_msecs": "1681159624000",
        "is_edit_eligible": true,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "82",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Mon Apr 10 20:17:04 +0000 2023",
        "conversation_id_str": "1645521299136032769",
        "display_text_range": [
          0,
          110
        ],
        "entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/7jm5EkjzGx",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1645521299136032769/photo/1",
              "id_str": "1645521297001119763",
              "indices": [
                111,
                134
              ],
              "media_url_https": "https://pbs.twimg.com/media/FtYQ5roXgBMOnUH.png",
              "type": "photo",
              "url": "https://t.co/7jm5EkjzGx",
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "medium": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 416,
                "width": 796,
                "focus_rects": [
                  {
                    "x": 27,
                    "y": 0,
                    "w": 743,
                    "h": 416
                  },
                  {
                    "x": 190,
                    "y": 0,
                    "w": 416,
                    "h": 416
                  },
                  {
                    "x": 216,
                    "y": 0,
                    "w": 365,
                    "h": 416
                  },
                  {
                    "x": 294,
                    "y": 0,
                    "w": 208,
                    "h": 416
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 796,
                    "h": 416
                  }
                ]
              }
            }
          ],
          "user_mentions": [],
          "urls": [],
          "hashtags": [],
          "symbols": []
        },
        "extended_entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/7jm5EkjzGx",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1645521299136032769/photo/1",
              "id_str": "1645521297001119763",
              "indices": [
                111,
                134
              ],
              "media_key": "3_1645521297001119763",
              "media_url_https": "https://pbs.twimg.com/media/FtYQ5roXgBMOnUH.png",
              "type": "photo",
              "url": "https://t.co/7jm5EkjzGx",
              "ext_media_availability": {
                "status": "Available"
              },
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "medium": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 416,
                "width": 796,
                "focus_rects": [
                  {
                    "x": 27,
                    "y": 0,
                    "w": 743,
                    "h": 416
                  },
                  {
                    "x": 190,
                    "y": 0,
                    "w": 416,
                    "h": 416
                  },
                  {
                    "x": 216,
                    "y": 0,
                    "w": 365,
                    "h": 416
                  },
                  {
                    "x": 294,
                    "y": 0,
                    "w": 208,
                    "h": 416
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 796,
                    "h": 416
                  }
                ]
              }
            }
          ]
        },
        "favorite_count": 1,
        "favorited": false,
        "full_text": "A new blog post has been published! \n\nHow to Scrape Sitemaps to Discover Scraping Targets \ud83e\udd16\n\nCheckout it out \ud83d\udc47 https://t.co/7jm5EkjzGx",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1645521299136032769"
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1645490142763786260",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1645490142763786260"
        ],
        "editable_until_msecs": "1681152196000",
        "is_edit_eligible": true,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "62",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Mon Apr 10 18:13:16 +0000 2023",
        "conversation_id_str": "1645490142763786260",
        "display_text_range": [
          0,
          114
        ],
        "entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/8KqHNTH8Ar",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1645490142763786260/photo/1",
              "id_str": "1645490139924164608",
              "indices": [
                115,
                138
              ],
              "media_url_https": "https://pbs.twimg.com/media/FtX0kGeWcAAH-kz.png",
              "type": "photo",
              "url": "https://t.co/8KqHNTH8Ar",
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "medium": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 416,
                "width": 796,
                "focus_rects": [
                  {
                    "x": 27,
                    "y": 0,
                    "w": 743,
                    "h": 416
                  },
                  {
                    "x": 190,
                    "y": 0,
                    "w": 416,
                    "h": 416
                  },
                  {
                    "x": 216,
                    "y": 0,
                    "w": 365,
                    "h": 416
                  },
                  {
                    "x": 294,
                    "y": 0,
                    "w": 208,
                    "h": 416
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 796,
                    "h": 416
                  }
                ]
              }
            }
          ],
          "user_mentions": [],
          "urls": [],
          "hashtags": [],
          "symbols": []
        },
        "extended_entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/8KqHNTH8Ar",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1645490142763786260/photo/1",
              "id_str": "1645490139924164608",
              "indices": [
                115,
                138
              ],
              "media_key": "3_1645490139924164608",
              "media_url_https": "https://pbs.twimg.com/media/FtX0kGeWcAAH-kz.png",
              "type": "photo",
              "url": "https://t.co/8KqHNTH8Ar",
              "ext_media_availability": {
                "status": "Available"
              },
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "medium": {
                  "h": 416,
                  "w": 796,
                  "resize": "fit"
                },
                "small": {
                  "h": 355,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 416,
                "width": 796,
                "focus_rects": [
                  {
                    "x": 27,
                    "y": 0,
                    "w": 743,
                    "h": 416
                  },
                  {
                    "x": 190,
                    "y": 0,
                    "w": 416,
                    "h": 416
                  },
                  {
                    "x": 216,
                    "y": 0,
                    "w": 365,
                    "h": 416
                  },
                  {
                    "x": 294,
                    "y": 0,
                    "w": 208,
                    "h": 416
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 796,
                    "h": 416
                  }
                ]
              }
            }
          ]
        },
        "favorite_count": 0,
        "favorited": false,
        "full_text": "A new blog post has been published! \n\nHow to Scrape Fashionphile for Second Hand Fashion Data \ud83e\udd16\n\nCheckout it out \ud83d\udc47 https://t.co/8KqHNTH8Ar",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1645490142763786260"
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1628001040154128390",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "card": {
        "rest_id": "https://t.co/x6EHAZFElO",
        "legacy": {
          "binding_values": [
            {
              "key": "photo_image_full_size_large",
              "value": {
                "image_value": {
                  "height": 419,
                  "width": 800,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=800x419"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image",
              "value": {
                "image_value": {
                  "height": 146,
                  "width": 280,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=280x150"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "description",
              "value": {
                "string_value": "Intro to using Python and JSONPath library and a query language for parsing JSON datasets.",
                "type": "STRING"
              }
            },
            {
              "key": "domain",
              "value": {
                "string_value": "scrapfly.io",
                "type": "STRING"
              }
            },
            {
              "key": "thumbnail_image_large",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=600x600"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image_small",
              "value": {
                "image_value": {
                  "height": 202,
                  "width": 386,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=386x202"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_original",
              "value": {
                "image_value": {
                  "height": 1572,
                  "width": 3008,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=orig"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "site",
              "value": {
                "scribe_key": "publisher_id",
                "type": "USER",
                "user_value": {
                  "id_str": "1310623081300402178",
                  "path": []
                }
              }
            },
            {
              "key": "photo_image_full_size_small",
              "value": {
                "image_value": {
                  "height": 202,
                  "width": 386,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=386x202"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image_large",
              "value": {
                "image_value": {
                  "height": 419,
                  "width": 800,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=800x419"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_small",
              "value": {
                "image_value": {
                  "height": 75,
                  "width": 144,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=144x144"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_x_large",
              "value": {
                "image_value": {
                  "height": 1070,
                  "width": 2048,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "photo_image_full_size_original",
              "value": {
                "image_value": {
                  "height": 1572,
                  "width": 3008,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=orig"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "vanity_url",
              "value": {
                "scribe_key": "vanity_url",
                "string_value": "scrapfly.io",
                "type": "STRING"
              }
            },
            {
              "key": "photo_image_full_size",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=600x314"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 168,
                        "red": 77
                      },
                      "percentage": 98.78
                    },
                    {
                      "rgb": {
                        "blue": 249,
                        "green": 219,
                        "red": 182
                      },
                      "percentage": 1.22
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "title",
              "value": {
                "string_value": "Quick Intro to Parsing JSON with JSONPath in Python",
                "type": "STRING"
              }
            },
            {
              "key": "summary_photo_image_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 168,
                        "red": 77
                      },
                      "percentage": 98.78
                    },
                    {
                      "rgb": {
                        "blue": 249,
                        "green": 219,
                        "red": 182
                      },
                      "percentage": 1.22
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "summary_photo_image_x_large",
              "value": {
                "image_value": {
                  "height": 1070,
                  "width": 2048,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=600x314"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "photo_image_full_size_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 168,
                        "red": 77
                      },
                      "percentage": 98.78
                    },
                    {
                      "rgb": {
                        "blue": 249,
                        "green": 219,
                        "red": 182
                      },
                      "percentage": 1.22
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "photo_image_full_size_x_large",
              "value": {
                "image_value": {
                  "height": 1070,
                  "width": 2048,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "card_url",
              "value": {
                "scribe_key": "card_url",
                "string_value": "https://t.co/x6EHAZFElO",
                "type": "STRING"
              }
            },
            {
              "key": "summary_photo_image_original",
              "value": {
                "image_value": {
                  "height": 1572,
                  "width": 3008,
                  "url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=orig"
                },
                "type": "IMAGE"
              }
            }
          ],
          "card_platform": {
            "platform": {
              "audience": {
                "name": "production"
              },
              "device": {
                "name": "Swift",
                "version": "12"
              }
            }
          },
          "name": "summary_large_image",
          "url": "https://t.co/x6EHAZFElO",
          "user_refs_results": [
            {
              "result": {
                "__typename": "User",
                "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
                "rest_id": "1310623081300402178",
                "affiliates_highlighted_label": {},
                "is_blue_verified": true,
                "profile_image_shape": "Circle",
                "legacy": {
                  "created_at": "Mon Sep 28 16:51:22 +0000 2020",
                  "default_profile": true,
                  "default_profile_image": false,
                  "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
                  "entities": {
                    "description": {
                      "urls": []
                    },
                    "url": {
                      "urls": [
                        {
                          "display_url": "scrapfly.io",
                          "expanded_url": "https://scrapfly.io",
                          "url": "https://t.co/1Is3k6KzyM",
                          "indices": [
                            0,
                            23
                          ]
                        }
                      ]
                    }
                  },
                  "fast_followers_count": 0,
                  "favourites_count": 26,
                  "followers_count": 163,
                  "friends_count": 993,
                  "has_custom_timelines": true,
                  "is_translator": false,
                  "listed_count": 2,
                  "location": "Paris",
                  "media_count": 11,
                  "name": "Scrapfly",
                  "normal_followers_count": 163,
                  "pinned_tweet_ids_str": [],
                  "possibly_sensitive": false,
                  "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
                  "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
                  "profile_interstitial_type": "",
                  "screen_name": "Scrapfly_dev",
                  "statuses_count": 56,
                  "translator_type": "none",
                  "url": "https://t.co/1Is3k6KzyM",
                  "verified": false,
                  "withheld_in_countries": []
                }
              }
            }
          ]
        }
      },
      "unified_card": {
        "card_fetch_state": "NoCard"
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1628001040154128390"
        ],
        "editable_until_msecs": "1676982469000",
        "is_edit_eligible": false,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "64",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Tue Feb 21 11:57:49 +0000 2023",
        "conversation_id_str": "1628001035011915778",
        "display_text_range": [
          0,
          99
        ],
        "entities": {
          "user_mentions": [],
          "urls": [
            {
              "display_url": "scrapfly.io/blog/parse-jso\u2026",
              "expanded_url": "https://scrapfly.io/blog/parse-json-jsonpath-python",
              "url": "https://t.co/x6EHAZFElO",
              "indices": [
                76,
                99
              ]
            }
          ],
          "hashtags": [
            {
              "indices": [
                56,
                68
              ],
              "text": "webScraping"
            }
          ],
          "symbols": []
        },
        "favorite_count": 0,
        "favorited": false,
        "full_text": "For more see our full intro and how JSONPath is used in #webScraping here:\n\nhttps://t.co/x6EHAZFElO",
        "in_reply_to_screen_name": "Scrapfly_dev",
        "in_reply_to_status_id_str": "1628001037750784001",
        "in_reply_to_user_id_str": "1310623081300402178",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1628001040154128390",
        "self_thread": {
          "id_str": "1628001035011915778"
        }
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1628001037750784001",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1628001037750784001"
        ],
        "editable_until_msecs": "1676982468000",
        "is_edit_eligible": false,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "92",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Tue Feb 21 11:57:48 +0000 2023",
        "conversation_id_str": "1628001035011915778",
        "display_text_range": [
          0,
          275
        ],
        "entities": {
          "user_mentions": [],
          "urls": [],
          "hashtags": [
            {
              "indices": [
                24,
                30
              ],
              "text": "XPath"
            }
          ],
          "symbols": []
        },
        "favorite_count": 0,
        "favorited": false,
        "full_text": "JSONPath is inspired by #XPath - a popular HTML parsing path language so the use is very similar.\n\nYou can easily query complex JSON datasets with short, recursive queries like `$..product[price&gt;20]` will select all products with price&gt;20 found ANYWHERE in the dataset.",
        "in_reply_to_screen_name": "Scrapfly_dev",
        "in_reply_to_status_id_str": "1628001035011915778",
        "in_reply_to_user_id_str": "1310623081300402178",
        "is_quote_status": false,
        "lang": "en",
        "quote_count": 0,
        "reply_count": 1,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1628001037750784001",
        "self_thread": {
          "id_str": "1628001035011915778"
        }
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1628001035011915778",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1628001035011915778"
        ],
        "editable_until_msecs": "1676982468000",
        "is_edit_eligible": false,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "148",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Tue Feb 21 11:57:48 +0000 2023",
        "conversation_id_str": "1628001035011915778",
        "display_text_range": [
          0,
          172
        ],
        "entities": {
          "user_mentions": [],
          "urls": [],
          "hashtags": [
            {
              "indices": [
                0,
                9
              ],
              "text": "jsonpath"
            },
            {
              "indices": [
                100,
                107
              ],
              "text": "python"
            },
            {
              "indices": [
                109,
                120
              ],
              "text": "javascript"
            },
            {
              "indices": [
                122,
                127
              ],
              "text": "ruby"
            },
            {
              "indices": [
                129,
                131
              ],
              "text": "r"
            },
            {
              "indices": [
                136,
                139
              ],
              "text": "go"
            },
            {
              "indices": [
                159,
                171
              ],
              "text": "webScraping"
            }
          ],
          "symbols": []
        },
        "favorite_count": 0,
        "favorited": false,
        "full_text": "#jsonpath is a path language for parsing JSON datasets and it's pretty awesome!  \nIt's available in #python, #javascript, #ruby, #r and #go and is perfect for #webScraping\ud83e\uddf5",
        "is_quote_status": false,
        "lang": "en",
        "quote_count": 0,
        "reply_count": 1,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1628001035011915778",
        "self_thread": {
          "id_str": "1628001035011915778"
        }
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1625776258146795520",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "card": {
        "rest_id": "https://t.co/mJjtOXGh57",
        "legacy": {
          "binding_values": [
            {
              "key": "photo_image_full_size_large",
              "value": {
                "image_value": {
                  "height": 419,
                  "width": 800,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=800x419"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image",
              "value": {
                "image_value": {
                  "height": 146,
                  "width": 280,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=280x150"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "description",
              "value": {
                "string_value": "The visible HTML doesn't always represent the whole dataset available on the page. In this article, we'll be taking a look at scraping of hidden web data. What is it and how can we scrape it using...",
                "type": "STRING"
              }
            },
            {
              "key": "domain",
              "value": {
                "string_value": "scrapfly.io",
                "type": "STRING"
              }
            },
            {
              "key": "thumbnail_image_large",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=600x600"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image_small",
              "value": {
                "image_value": {
                  "height": 202,
                  "width": 386,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=386x202"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_original",
              "value": {
                "image_value": {
                  "height": 1572,
                  "width": 3008,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "site",
              "value": {
                "scribe_key": "publisher_id",
                "type": "USER",
                "user_value": {
                  "id_str": "1310623081300402178",
                  "path": []
                }
              }
            },
            {
              "key": "photo_image_full_size_small",
              "value": {
                "image_value": {
                  "height": 202,
                  "width": 386,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=386x202"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image_large",
              "value": {
                "image_value": {
                  "height": 419,
                  "width": 800,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=800x419"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_small",
              "value": {
                "image_value": {
                  "height": 75,
                  "width": 144,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=144x144"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_x_large",
              "value": {
                "image_value": {
                  "height": 1070,
                  "width": 2048,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "photo_image_full_size_original",
              "value": {
                "image_value": {
                  "height": 1572,
                  "width": 3008,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "vanity_url",
              "value": {
                "scribe_key": "vanity_url",
                "string_value": "scrapfly.io",
                "type": "STRING"
              }
            },
            {
              "key": "photo_image_full_size",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=600x314"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 168,
                        "red": 78
                      },
                      "percentage": 98.74
                    },
                    {
                      "rgb": {
                        "blue": 248,
                        "green": 216,
                        "red": 176
                      },
                      "percentage": 1.26
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "title",
              "value": {
                "string_value": "How to Scrape Hidden Web Data",
                "type": "STRING"
              }
            },
            {
              "key": "summary_photo_image_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 168,
                        "red": 78
                      },
                      "percentage": 98.74
                    },
                    {
                      "rgb": {
                        "blue": 248,
                        "green": 216,
                        "red": 176
                      },
                      "percentage": 1.26
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "summary_photo_image_x_large",
              "value": {
                "image_value": {
                  "height": 1070,
                  "width": 2048,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=600x314"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "photo_image_full_size_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 168,
                        "red": 78
                      },
                      "percentage": 98.74
                    },
                    {
                      "rgb": {
                        "blue": 248,
                        "green": 216,
                        "red": 176
                      },
                      "percentage": 1.26
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "photo_image_full_size_x_large",
              "value": {
                "image_value": {
                  "height": 1070,
                  "width": 2048,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "card_url",
              "value": {
                "scribe_key": "card_url",
                "string_value": "https://t.co/mJjtOXGh57",
                "type": "STRING"
              }
            },
            {
              "key": "summary_photo_image_original",
              "value": {
                "image_value": {
                  "height": 1572,
                  "width": 3008,
                  "url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            }
          ],
          "card_platform": {
            "platform": {
              "audience": {
                "name": "production"
              },
              "device": {
                "name": "Swift",
                "version": "12"
              }
            }
          },
          "name": "summary_large_image",
          "url": "https://t.co/mJjtOXGh57",
          "user_refs_results": [
            {
              "result": {
                "__typename": "User",
                "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
                "rest_id": "1310623081300402178",
                "affiliates_highlighted_label": {},
                "is_blue_verified": true,
                "profile_image_shape": "Circle",
                "legacy": {
                  "created_at": "Mon Sep 28 16:51:22 +0000 2020",
                  "default_profile": true,
                  "default_profile_image": false,
                  "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
                  "entities": {
                    "description": {
                      "urls": []
                    },
                    "url": {
                      "urls": [
                        {
                          "display_url": "scrapfly.io",
                          "expanded_url": "https://scrapfly.io",
                          "url": "https://t.co/1Is3k6KzyM",
                          "indices": [
                            0,
                            23
                          ]
                        }
                      ]
                    }
                  },
                  "fast_followers_count": 0,
                  "favourites_count": 26,
                  "followers_count": 163,
                  "friends_count": 993,
                  "has_custom_timelines": true,
                  "is_translator": false,
                  "listed_count": 2,
                  "location": "Paris",
                  "media_count": 11,
                  "name": "Scrapfly",
                  "normal_followers_count": 163,
                  "pinned_tweet_ids_str": [],
                  "possibly_sensitive": false,
                  "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
                  "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
                  "profile_interstitial_type": "",
                  "screen_name": "Scrapfly_dev",
                  "statuses_count": 56,
                  "translator_type": "none",
                  "url": "https://t.co/1Is3k6KzyM",
                  "verified": false,
                  "withheld_in_countries": []
                }
              }
            }
          ]
        }
      },
      "unified_card": {
        "card_fetch_state": "NoCard"
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1625776258146795520"
        ],
        "editable_until_msecs": "1676452040000",
        "is_edit_eligible": false,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "60",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Wed Feb 15 08:37:20 +0000 2023",
        "conversation_id_str": "1625775862946893824",
        "display_text_range": [
          0,
          248
        ],
        "entities": {
          "user_mentions": [],
          "urls": [
            {
              "display_url": "scrapfly.io/blog/how-to-sc\u2026",
              "expanded_url": "https://scrapfly.io/blog/how-to-scrape-hidden-web-data/",
              "url": "https://t.co/mJjtOXGh57",
              "indices": [
                225,
                248
              ]
            }
          ],
          "hashtags": [
            {
              "indices": [
                76,
                85
              ],
              "text": "selenium"
            },
            {
              "indices": [
                87,
                97
              ],
              "text": "puppeteer"
            },
            {
              "indices": [
                101,
                112
              ],
              "text": "playwright"
            },
            {
              "indices": [
                190,
                197
              ],
              "text": "python"
            }
          ],
          "symbols": []
        },
        "favorite_count": 1,
        "favorited": false,
        "full_text": "So, to scrape hidden web data we don't need to use a real web browser using #selenium, #puppeteer or #playwright. We can grab the javacript variable dataset by parsing the script with basic #python!\n\nSee our full guide here: https://t.co/mJjtOXGh57",
        "in_reply_to_screen_name": "Scrapfly_dev",
        "in_reply_to_status_id_str": "1625776006555635714",
        "in_reply_to_user_id_str": "1310623081300402178",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 1,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1625776258146795520",
        "self_thread": {
          "id_str": "1625775862946893824"
        }
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1625776006555635714",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1625776006555635714"
        ],
        "editable_until_msecs": "1676451980000",
        "is_edit_eligible": false,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "73",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Wed Feb 15 08:36:20 +0000 2023",
        "conversation_id_str": "1625775862946893824",
        "display_text_range": [
          0,
          88
        ],
        "entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/Ew4p5B9CEJ",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1625776006555635714/photo/1",
              "id_str": "1625775987895177216",
              "indices": [
                89,
                112
              ],
              "media_url_https": "https://pbs.twimg.com/media/Fo_qpbLaEAARHcN.jpg",
              "type": "photo",
              "url": "https://t.co/Ew4p5B9CEJ",
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 567,
                  "w": 1000,
                  "resize": "fit"
                },
                "medium": {
                  "h": 567,
                  "w": 1000,
                  "resize": "fit"
                },
                "small": {
                  "h": 386,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 567,
                "width": 1000,
                "focus_rects": [
                  {
                    "x": 0,
                    "y": 0,
                    "w": 1000,
                    "h": 560
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 567,
                    "h": 567
                  },
                  {
                    "x": 27,
                    "y": 0,
                    "w": 497,
                    "h": 567
                  },
                  {
                    "x": 133,
                    "y": 0,
                    "w": 284,
                    "h": 567
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 1000,
                    "h": 567
                  }
                ]
              }
            }
          ],
          "user_mentions": [],
          "urls": [],
          "hashtags": [],
          "symbols": []
        },
        "extended_entities": {
          "media": [
            {
              "display_url": "pic.twitter.com/Ew4p5B9CEJ",
              "expanded_url": "https://twitter.com/Scrapfly_dev/status/1625776006555635714/photo/1",
              "id_str": "1625775987895177216",
              "indices": [
                89,
                112
              ],
              "media_key": "3_1625775987895177216",
              "media_url_https": "https://pbs.twimg.com/media/Fo_qpbLaEAARHcN.jpg",
              "type": "photo",
              "url": "https://t.co/Ew4p5B9CEJ",
              "ext_media_availability": {
                "status": "Available"
              },
              "features": {
                "large": {
                  "faces": []
                },
                "medium": {
                  "faces": []
                },
                "small": {
                  "faces": []
                },
                "orig": {
                  "faces": []
                }
              },
              "sizes": {
                "large": {
                  "h": 567,
                  "w": 1000,
                  "resize": "fit"
                },
                "medium": {
                  "h": 567,
                  "w": 1000,
                  "resize": "fit"
                },
                "small": {
                  "h": 386,
                  "w": 680,
                  "resize": "fit"
                },
                "thumb": {
                  "h": 150,
                  "w": 150,
                  "resize": "crop"
                }
              },
              "original_info": {
                "height": 567,
                "width": 1000,
                "focus_rects": [
                  {
                    "x": 0,
                    "y": 0,
                    "w": 1000,
                    "h": 560
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 567,
                    "h": 567
                  },
                  {
                    "x": 27,
                    "y": 0,
                    "w": 497,
                    "h": 567
                  },
                  {
                    "x": 133,
                    "y": 0,
                    "w": 284,
                    "h": 567
                  },
                  {
                    "x": 0,
                    "y": 0,
                    "w": 1000,
                    "h": 567
                  }
                ]
              }
            }
          ]
        },
        "favorite_count": 0,
        "favorited": false,
        "full_text": "For example, here's how Zillow stores it's real estate property data in the page source: https://t.co/Ew4p5B9CEJ",
        "in_reply_to_screen_name": "Scrapfly_dev",
        "in_reply_to_status_id_str": "1625775929338515456",
        "in_reply_to_user_id_str": "1310623081300402178",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 1,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1625776006555635714",
        "self_thread": {
          "id_str": "1625775862946893824"
        }
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1625775929338515456",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1625775929338515456"
        ],
        "editable_until_msecs": "1676451961000",
        "is_edit_eligible": false,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "52",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Wed Feb 15 08:36:01 +0000 2023",
        "conversation_id_str": "1625775862946893824",
        "display_text_range": [
          0,
          270
        ],
        "entities": {
          "user_mentions": [],
          "urls": [],
          "hashtags": [
            {
              "indices": [
                242,
                250
              ],
              "text": "reactjs"
            },
            {
              "indices": [
                251,
                257
              ],
              "text": "vuejs"
            },
            {
              "indices": [
                262,
                269
              ],
              "text": "nextjs"
            }
          ],
          "symbols": []
        },
        "favorite_count": 0,
        "favorited": false,
        "full_text": "Modern javascript-powered websites often use js variables to store page data. \nThis means scrapers without web browser capabilities cannot see the data in the HTML. Most notably this technique is used by popular javascript frameworks such as #reactjs #vuejs and #nextjs.",
        "in_reply_to_screen_name": "Scrapfly_dev",
        "in_reply_to_status_id_str": "1625775862946893824",
        "in_reply_to_user_id_str": "1310623081300402178",
        "is_quote_status": false,
        "lang": "en",
        "quote_count": 0,
        "reply_count": 1,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1625775929338515456",
        "self_thread": {
          "id_str": "1625775862946893824"
        }
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1625775862946893824",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1625775862946893824"
        ],
        "editable_until_msecs": "1676451945000",
        "is_edit_eligible": false,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "127",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Wed Feb 15 08:35:45 +0000 2023",
        "conversation_id_str": "1625775862946893824",
        "display_text_range": [
          0,
          78
        ],
        "entities": {
          "user_mentions": [],
          "urls": [],
          "hashtags": [
            {
              "indices": [
                54,
                66
              ],
              "text": "webScraping"
            }
          ],
          "symbols": []
        },
        "favorite_count": 1,
        "favorited": false,
        "full_text": "Web scraping hidden web data is becoming the number 1 #webScraping technique \ud83e\uddf5",
        "is_quote_status": false,
        "lang": "en",
        "quote_count": 0,
        "reply_count": 1,
        "retweet_count": 1,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1625775862946893824",
        "self_thread": {
          "id_str": "1625775862946893824"
        }
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1625137012776001538",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "card": {
        "rest_id": "https://t.co/2kiOGlfKPY",
        "legacy": {
          "binding_values": [
            {
              "key": "photo_image_full_size_large",
              "value": {
                "image_value": {
                  "height": 416,
                  "width": 794,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=800x419"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image",
              "value": {
                "image_value": {
                  "height": 146,
                  "width": 280,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=280x150"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "description",
              "value": {
                "string_value": "With the news of Twitter dropping free API access we're taking a look at web scraping Twitter using Python for free. In this tutorial we'll cover two methods: using Playwright and Twitter's hidden...",
                "type": "STRING"
              }
            },
            {
              "key": "domain",
              "value": {
                "string_value": "scrapfly.io",
                "type": "STRING"
              }
            },
            {
              "key": "thumbnail_image_large",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=600x600"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image_small",
              "value": {
                "image_value": {
                  "height": 202,
                  "width": 386,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=386x202"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_original",
              "value": {
                "image_value": {
                  "height": 416,
                  "width": 796,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "site",
              "value": {
                "scribe_key": "publisher_id",
                "type": "USER",
                "user_value": {
                  "id_str": "1310623081300402178",
                  "path": []
                }
              }
            },
            {
              "key": "photo_image_full_size_small",
              "value": {
                "image_value": {
                  "height": 202,
                  "width": 386,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=386x202"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image_large",
              "value": {
                "image_value": {
                  "height": 416,
                  "width": 794,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=800x419"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_small",
              "value": {
                "image_value": {
                  "height": 75,
                  "width": 144,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=144x144"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_x_large",
              "value": {
                "image_value": {
                  "height": 416,
                  "width": 796,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "photo_image_full_size_original",
              "value": {
                "image_value": {
                  "height": 416,
                  "width": 796,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "vanity_url",
              "value": {
                "scribe_key": "vanity_url",
                "string_value": "scrapfly.io",
                "type": "STRING"
              }
            },
            {
              "key": "photo_image_full_size",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=600x314"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 168,
                        "red": 78
                      },
                      "percentage": 95.42
                    },
                    {
                      "rgb": {
                        "blue": 253,
                        "green": 244,
                        "red": 232
                      },
                      "percentage": 4.58
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "title",
              "value": {
                "string_value": "How to Scrape Twitter with Python",
                "type": "STRING"
              }
            },
            {
              "key": "summary_photo_image_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 168,
                        "red": 78
                      },
                      "percentage": 95.42
                    },
                    {
                      "rgb": {
                        "blue": 253,
                        "green": 244,
                        "red": 232
                      },
                      "percentage": 4.58
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "summary_photo_image_x_large",
              "value": {
                "image_value": {
                  "height": 416,
                  "width": 796,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=600x314"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "photo_image_full_size_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 168,
                        "red": 78
                      },
                      "percentage": 95.42
                    },
                    {
                      "rgb": {
                        "blue": 253,
                        "green": 244,
                        "red": 232
                      },
                      "percentage": 4.58
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "photo_image_full_size_x_large",
              "value": {
                "image_value": {
                  "height": 416,
                  "width": 796,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "card_url",
              "value": {
                "scribe_key": "card_url",
                "string_value": "https://t.co/2kiOGlfKPY",
                "type": "STRING"
              }
            },
            {
              "key": "summary_photo_image_original",
              "value": {
                "image_value": {
                  "height": 416,
                  "width": 796,
                  "url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            }
          ],
          "card_platform": {
            "platform": {
              "audience": {
                "name": "production"
              },
              "device": {
                "name": "Swift",
                "version": "12"
              }
            }
          },
          "name": "summary_large_image",
          "url": "https://t.co/2kiOGlfKPY",
          "user_refs_results": [
            {
              "result": {
                "__typename": "User",
                "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
                "rest_id": "1310623081300402178",
                "affiliates_highlighted_label": {},
                "is_blue_verified": true,
                "profile_image_shape": "Circle",
                "legacy": {
                  "created_at": "Mon Sep 28 16:51:22 +0000 2020",
                  "default_profile": true,
                  "default_profile_image": false,
                  "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
                  "entities": {
                    "description": {
                      "urls": []
                    },
                    "url": {
                      "urls": [
                        {
                          "display_url": "scrapfly.io",
                          "expanded_url": "https://scrapfly.io",
                          "url": "https://t.co/1Is3k6KzyM",
                          "indices": [
                            0,
                            23
                          ]
                        }
                      ]
                    }
                  },
                  "fast_followers_count": 0,
                  "favourites_count": 26,
                  "followers_count": 163,
                  "friends_count": 993,
                  "has_custom_timelines": true,
                  "is_translator": false,
                  "listed_count": 2,
                  "location": "Paris",
                  "media_count": 11,
                  "name": "Scrapfly",
                  "normal_followers_count": 163,
                  "pinned_tweet_ids_str": [],
                  "possibly_sensitive": false,
                  "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
                  "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
                  "profile_interstitial_type": "",
                  "screen_name": "Scrapfly_dev",
                  "statuses_count": 56,
                  "translator_type": "none",
                  "url": "https://t.co/1Is3k6KzyM",
                  "verified": false,
                  "withheld_in_countries": []
                }
              }
            }
          ]
        }
      },
      "unified_card": {
        "card_fetch_state": "NoCard"
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1625137012776001538"
        ],
        "editable_until_msecs": "1676299632000",
        "is_edit_eligible": true,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "count": "298",
        "state": "EnabledWithCount"
      },
      "source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
      "legacy": {
        "bookmark_count": 4,
        "bookmarked": false,
        "created_at": "Mon Feb 13 14:17:12 +0000 2023",
        "conversation_id_str": "1625137012776001538",
        "display_text_range": [
          0,
          266
        ],
        "entities": {
          "user_mentions": [],
          "urls": [
            {
              "display_url": "scrapfly.io/blog/how-to-sc\u2026",
              "expanded_url": "https://scrapfly.io/blog/how-to-scrape-twitter/",
              "url": "https://t.co/2kiOGlfKPY",
              "indices": [
                229,
                252
              ]
            }
          ],
          "hashtags": [
            {
              "indices": [
                33,
                41
              ],
              "text": "Twitter"
            },
            {
              "indices": [
                57,
                64
              ],
              "text": "Python"
            },
            {
              "indices": [
                254,
                266
              ],
              "text": "webscraping"
            }
          ],
          "symbols": []
        },
        "favorite_count": 2,
        "favorited": false,
        "full_text": "Did you know that you can scrape #Twitter for free using #Python? \nTwitter is disabling free API access but that shouldn't stop you from getting that public data for your projects!\n\nWe cover 2 ways to scrape Twitter with Python:\nhttps://t.co/2kiOGlfKPY\n\n#webscraping",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 1,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1625137012776001538"
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1592574652445954049",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "card": {
        "rest_id": "https://t.co/4LlOIGwawB",
        "legacy": {
          "binding_values": [
            {
              "key": "photo_image_full_size_large",
              "value": {
                "image_value": {
                  "height": 419,
                  "width": 800,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=800x419"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image",
              "value": {
                "image_value": {
                  "height": 146,
                  "width": 280,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=280x150"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "description",
              "value": {
                "string_value": "Residential proxies are the most popular type of proxies used in web scraping. What makes a good residential proxy and what providers are the best?",
                "type": "STRING"
              }
            },
            {
              "key": "domain",
              "value": {
                "string_value": "scrapfly.io",
                "type": "STRING"
              }
            },
            {
              "key": "thumbnail_image_large",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=600x600"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image_small",
              "value": {
                "image_value": {
                  "height": 202,
                  "width": 386,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=386x202"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_original",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "site",
              "value": {
                "scribe_key": "publisher_id",
                "type": "USER",
                "user_value": {
                  "id_str": "1310623081300402178",
                  "path": []
                }
              }
            },
            {
              "key": "photo_image_full_size_small",
              "value": {
                "image_value": {
                  "height": 202,
                  "width": 386,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=386x202"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image_large",
              "value": {
                "image_value": {
                  "height": 419,
                  "width": 800,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=800x419"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_small",
              "value": {
                "image_value": {
                  "height": 75,
                  "width": 144,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=144x144"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_x_large",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "photo_image_full_size_original",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "vanity_url",
              "value": {
                "scribe_key": "vanity_url",
                "string_value": "scrapfly.io",
                "type": "STRING"
              }
            },
            {
              "key": "photo_image_full_size",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=600x314"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 171,
                        "red": 86
                      },
                      "percentage": 98.26
                    },
                    {
                      "rgb": {
                        "blue": 247,
                        "green": 212,
                        "red": 170
                      },
                      "percentage": 1.74
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "title",
              "value": {
                "string_value": "Top 5 Residential Proxy Providers for Web Scraping",
                "type": "STRING"
              }
            },
            {
              "key": "summary_photo_image_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 171,
                        "red": 86
                      },
                      "percentage": 98.26
                    },
                    {
                      "rgb": {
                        "blue": 247,
                        "green": 212,
                        "red": 170
                      },
                      "percentage": 1.74
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "summary_photo_image_x_large",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=600x314"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "photo_image_full_size_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 171,
                        "red": 86
                      },
                      "percentage": 98.26
                    },
                    {
                      "rgb": {
                        "blue": 247,
                        "green": 212,
                        "red": 170
                      },
                      "percentage": 1.74
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "photo_image_full_size_x_large",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "card_url",
              "value": {
                "scribe_key": "card_url",
                "string_value": "https://t.co/4LlOIGwawB",
                "type": "STRING"
              }
            },
            {
              "key": "summary_photo_image_original",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            }
          ],
          "card_platform": {
            "platform": {
              "audience": {
                "name": "production"
              },
              "device": {
                "name": "Swift",
                "version": "12"
              }
            }
          },
          "name": "summary_large_image",
          "url": "https://t.co/4LlOIGwawB",
          "user_refs_results": [
            {
              "result": {
                "__typename": "User",
                "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
                "rest_id": "1310623081300402178",
                "affiliates_highlighted_label": {},
                "is_blue_verified": true,
                "profile_image_shape": "Circle",
                "legacy": {
                  "created_at": "Mon Sep 28 16:51:22 +0000 2020",
                  "default_profile": true,
                  "default_profile_image": false,
                  "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
                  "entities": {
                    "description": {
                      "urls": []
                    },
                    "url": {
                      "urls": [
                        {
                          "display_url": "scrapfly.io",
                          "expanded_url": "https://scrapfly.io",
                          "url": "https://t.co/1Is3k6KzyM",
                          "indices": [
                            0,
                            23
                          ]
                        }
                      ]
                    }
                  },
                  "fast_followers_count": 0,
                  "favourites_count": 26,
                  "followers_count": 163,
                  "friends_count": 993,
                  "has_custom_timelines": true,
                  "is_translator": false,
                  "listed_count": 2,
                  "location": "Paris",
                  "media_count": 11,
                  "name": "Scrapfly",
                  "normal_followers_count": 163,
                  "pinned_tweet_ids_str": [],
                  "possibly_sensitive": false,
                  "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
                  "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
                  "profile_interstitial_type": "",
                  "screen_name": "Scrapfly_dev",
                  "statuses_count": 56,
                  "translator_type": "none",
                  "url": "https://t.co/1Is3k6KzyM",
                  "verified": false,
                  "withheld_in_countries": []
                }
              }
            }
          ]
        }
      },
      "unified_card": {
        "card_fetch_state": "NoCard"
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1592574652445954049"
        ],
        "editable_until_msecs": "1668536160000",
        "is_edit_eligible": true,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "state": "Enabled"
      },
      "source": "<a href=\"https://www.semrush.com/\" rel=\"nofollow\">Semrush Social Media Tool</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Tue Nov 15 17:46:00 +0000 2022",
        "conversation_id_str": "1592574652445954049",
        "display_text_range": [
          0,
          111
        ],
        "entities": {
          "user_mentions": [],
          "urls": [
            {
              "display_url": "bit.ly/3yrRild",
              "expanded_url": "https://bit.ly/3yrRild",
              "url": "https://t.co/4LlOIGwawB",
              "indices": [
                88,
                111
              ]
            }
          ],
          "hashtags": [
            {
              "indices": [
                56,
                68
              ],
              "text": "webscraping"
            },
            {
              "indices": [
                69,
                86
              ],
              "text": "residentialproxy"
            }
          ],
          "symbols": []
        },
        "favorite_count": 0,
        "favorited": false,
        "full_text": "Top 5 Residential Proxy Providers for Web Scraping \ud83c\udf10 \ud83e\udd16\n\n#webscraping #residentialproxy\n\nhttps://t.co/4LlOIGwawB",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1592574652445954049"
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1589966079513673728",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "card": {
        "rest_id": "https://t.co/3ooXfi4xpx",
        "legacy": {
          "binding_values": [
            {
              "key": "photo_image_full_size_large",
              "value": {
                "image_value": {
                  "height": 419,
                  "width": 800,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=800x419"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image",
              "value": {
                "image_value": {
                  "height": 146,
                  "width": 280,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=280x150"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "description",
              "value": {
                "string_value": "Analysis and comparison of some of the most popular proxy providers. What makes a good proxy providers? What features and dangers to look out for?",
                "type": "STRING"
              }
            },
            {
              "key": "domain",
              "value": {
                "string_value": "scrapfly.io",
                "type": "STRING"
              }
            },
            {
              "key": "thumbnail_image_large",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=600x600"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image_small",
              "value": {
                "image_value": {
                  "height": 202,
                  "width": 386,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=386x202"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_original",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "site",
              "value": {
                "scribe_key": "publisher_id",
                "type": "USER",
                "user_value": {
                  "id_str": "1310623081300402178",
                  "path": []
                }
              }
            },
            {
              "key": "photo_image_full_size_small",
              "value": {
                "image_value": {
                  "height": 202,
                  "width": 386,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=386x202"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image_large",
              "value": {
                "image_value": {
                  "height": 419,
                  "width": 800,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=800x419"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_small",
              "value": {
                "image_value": {
                  "height": 75,
                  "width": 144,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=144x144"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_x_large",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "photo_image_full_size_original",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "vanity_url",
              "value": {
                "scribe_key": "vanity_url",
                "string_value": "scrapfly.io",
                "type": "STRING"
              }
            },
            {
              "key": "photo_image_full_size",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=600x314"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 171,
                        "red": 86
                      },
                      "percentage": 98.64
                    },
                    {
                      "rgb": {
                        "blue": 247,
                        "green": 212,
                        "red": 170
                      },
                      "percentage": 1.36
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "title",
              "value": {
                "string_value": "Best Proxy Providers for Web Scraping (2023 Update)",
                "type": "STRING"
              }
            },
            {
              "key": "summary_photo_image_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 171,
                        "red": 86
                      },
                      "percentage": 98.64
                    },
                    {
                      "rgb": {
                        "blue": 247,
                        "green": 212,
                        "red": 170
                      },
                      "percentage": 1.36
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "summary_photo_image_x_large",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=600x314"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "photo_image_full_size_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 171,
                        "red": 86
                      },
                      "percentage": 98.64
                    },
                    {
                      "rgb": {
                        "blue": 247,
                        "green": 212,
                        "red": 170
                      },
                      "percentage": 1.36
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "photo_image_full_size_x_large",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "card_url",
              "value": {
                "scribe_key": "card_url",
                "string_value": "https://t.co/3ooXfi4xpx",
                "type": "STRING"
              }
            },
            {
              "key": "summary_photo_image_original",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            }
          ],
          "card_platform": {
            "platform": {
              "audience": {
                "name": "production"
              },
              "device": {
                "name": "Swift",
                "version": "12"
              }
            }
          },
          "name": "summary_large_image",
          "url": "https://t.co/3ooXfi4xpx",
          "user_refs_results": [
            {
              "result": {
                "__typename": "User",
                "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
                "rest_id": "1310623081300402178",
                "affiliates_highlighted_label": {},
                "is_blue_verified": true,
                "profile_image_shape": "Circle",
                "legacy": {
                  "created_at": "Mon Sep 28 16:51:22 +0000 2020",
                  "default_profile": true,
                  "default_profile_image": false,
                  "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
                  "entities": {
                    "description": {
                      "urls": []
                    },
                    "url": {
                      "urls": [
                        {
                          "display_url": "scrapfly.io",
                          "expanded_url": "https://scrapfly.io",
                          "url": "https://t.co/1Is3k6KzyM",
                          "indices": [
                            0,
                            23
                          ]
                        }
                      ]
                    }
                  },
                  "fast_followers_count": 0,
                  "favourites_count": 26,
                  "followers_count": 163,
                  "friends_count": 993,
                  "has_custom_timelines": true,
                  "is_translator": false,
                  "listed_count": 2,
                  "location": "Paris",
                  "media_count": 11,
                  "name": "Scrapfly",
                  "normal_followers_count": 163,
                  "pinned_tweet_ids_str": [],
                  "possibly_sensitive": false,
                  "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
                  "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
                  "profile_interstitial_type": "",
                  "screen_name": "Scrapfly_dev",
                  "statuses_count": 56,
                  "translator_type": "none",
                  "url": "https://t.co/1Is3k6KzyM",
                  "verified": false,
                  "withheld_in_countries": []
                }
              }
            }
          ]
        }
      },
      "unified_card": {
        "card_fetch_state": "NoCard"
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1589966079513673728"
        ],
        "editable_until_msecs": "1667914228000",
        "is_edit_eligible": true,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "state": "Enabled"
      },
      "source": "<a href=\"https://www.semrush.com/\" rel=\"nofollow\">Semrush Social Media Tool</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Tue Nov 08 13:00:28 +0000 2022",
        "conversation_id_str": "1589966079513673728",
        "display_text_range": [
          0,
          201
        ],
        "entities": {
          "user_mentions": [],
          "urls": [
            {
              "display_url": "bit.ly/3CodMVc",
              "expanded_url": "https://bit.ly/3CodMVc",
              "url": "https://t.co/3ooXfi4xpx",
              "indices": [
                178,
                201
              ]
            }
          ],
          "hashtags": [
            {
              "indices": [
                157,
                169
              ],
              "text": "webscraping"
            },
            {
              "indices": [
                170,
                176
              ],
              "text": "proxy"
            }
          ],
          "symbols": []
        },
        "favorite_count": 0,
        "favorited": false,
        "full_text": "Best Proxy Providers for Web Scraping \ud83e\udd16\n\nWhat do we cover in this article?\ud83d\udc47\n\n\ud83d\udd77 Quality Evaluation\n\ud83d\udd77 Pricing Evaluation\n\ud83d\udd77 Evaluation Methodology\n\ud83d\udd77 Providers\n\n#webscraping #proxy\n\nhttps://t.co/3ooXfi4xpx",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 1,
        "retweet_count": 0,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1589966079513673728"
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    },
    {
      "__typename": "Tweet",
      "rest_id": "1587431468141318146",
      "core": {
        "user_results": {
          "result": {
            "__typename": "User",
            "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
            "rest_id": "1310623081300402178",
            "affiliates_highlighted_label": {},
            "is_blue_verified": true,
            "profile_image_shape": "Circle",
            "legacy": {
              "created_at": "Mon Sep 28 16:51:22 +0000 2020",
              "default_profile": true,
              "default_profile_image": false,
              "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
              "entities": {
                "description": {
                  "urls": []
                },
                "url": {
                  "urls": [
                    {
                      "display_url": "scrapfly.io",
                      "expanded_url": "https://scrapfly.io",
                      "url": "https://t.co/1Is3k6KzyM",
                      "indices": [
                        0,
                        23
                      ]
                    }
                  ]
                }
              },
              "fast_followers_count": 0,
              "favourites_count": 26,
              "followers_count": 163,
              "friends_count": 993,
              "has_custom_timelines": true,
              "is_translator": false,
              "listed_count": 2,
              "location": "Paris",
              "media_count": 11,
              "name": "Scrapfly",
              "normal_followers_count": 163,
              "pinned_tweet_ids_str": [],
              "possibly_sensitive": false,
              "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
              "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
              "profile_interstitial_type": "",
              "screen_name": "Scrapfly_dev",
              "statuses_count": 56,
              "translator_type": "none",
              "url": "https://t.co/1Is3k6KzyM",
              "verified": false,
              "withheld_in_countries": []
            }
          }
        }
      },
      "card": {
        "rest_id": "https://t.co/JLCeZGeDtF",
        "legacy": {
          "binding_values": [
            {
              "key": "photo_image_full_size_large",
              "value": {
                "image_value": {
                  "height": 419,
                  "width": 800,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=800x419"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image",
              "value": {
                "image_value": {
                  "height": 146,
                  "width": 280,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=280x150"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "description",
              "value": {
                "string_value": "Introduction to proxy usage in web scraping. What types of proxies are there? How to evaluate proxy providers and avoid common issues.",
                "type": "STRING"
              }
            },
            {
              "key": "domain",
              "value": {
                "string_value": "scrapfly.io",
                "type": "STRING"
              }
            },
            {
              "key": "thumbnail_image_large",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=600x600"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image_small",
              "value": {
                "image_value": {
                  "height": 202,
                  "width": 386,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=386x202"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_original",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "site",
              "value": {
                "scribe_key": "publisher_id",
                "type": "USER",
                "user_value": {
                  "id_str": "1310623081300402178",
                  "path": []
                }
              }
            },
            {
              "key": "photo_image_full_size_small",
              "value": {
                "image_value": {
                  "height": 202,
                  "width": 386,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=386x202"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image_large",
              "value": {
                "image_value": {
                  "height": 419,
                  "width": 800,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=800x419"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_small",
              "value": {
                "image_value": {
                  "height": 75,
                  "width": 144,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=144x144"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_x_large",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "photo_image_full_size_original",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "vanity_url",
              "value": {
                "scribe_key": "vanity_url",
                "string_value": "scrapfly.io",
                "type": "STRING"
              }
            },
            {
              "key": "photo_image_full_size",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=600x314"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "thumbnail_image_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 171,
                        "red": 86
                      },
                      "percentage": 98.74
                    },
                    {
                      "rgb": {
                        "blue": 247,
                        "green": 211,
                        "red": 167
                      },
                      "percentage": 1.26
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "title",
              "value": {
                "string_value": "Introduction To Proxies in Web Scraping",
                "type": "STRING"
              }
            },
            {
              "key": "summary_photo_image_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 171,
                        "red": 86
                      },
                      "percentage": 98.74
                    },
                    {
                      "rgb": {
                        "blue": 247,
                        "green": 211,
                        "red": 167
                      },
                      "percentage": 1.26
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "summary_photo_image_x_large",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "summary_photo_image",
              "value": {
                "image_value": {
                  "height": 314,
                  "width": 600,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=600x314"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "photo_image_full_size_color",
              "value": {
                "image_color_value": {
                  "palette": [
                    {
                      "rgb": {
                        "blue": 241,
                        "green": 171,
                        "red": 86
                      },
                      "percentage": 98.74
                    },
                    {
                      "rgb": {
                        "blue": 247,
                        "green": 211,
                        "red": 167
                      },
                      "percentage": 1.26
                    }
                  ]
                },
                "type": "IMAGE_COLOR"
              }
            },
            {
              "key": "photo_image_full_size_x_large",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=2048x2048_2_exp"
                },
                "type": "IMAGE"
              }
            },
            {
              "key": "card_url",
              "value": {
                "scribe_key": "card_url",
                "string_value": "https://t.co/JLCeZGeDtF",
                "type": "STRING"
              }
            },
            {
              "key": "summary_photo_image_original",
              "value": {
                "image_value": {
                  "height": 430,
                  "width": 822,
                  "url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=orig"
                },
                "type": "IMAGE"
              }
            }
          ],
          "card_platform": {
            "platform": {
              "audience": {
                "name": "production"
              },
              "device": {
                "name": "Swift",
                "version": "12"
              }
            }
          },
          "name": "summary_large_image",
          "url": "https://t.co/JLCeZGeDtF",
          "user_refs_results": [
            {
              "result": {
                "__typename": "User",
                "id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
                "rest_id": "1310623081300402178",
                "affiliates_highlighted_label": {},
                "is_blue_verified": true,
                "profile_image_shape": "Circle",
                "legacy": {
                  "created_at": "Mon Sep 28 16:51:22 +0000 2020",
                  "default_profile": true,
                  "default_profile_image": false,
                  "description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
                  "entities": {
                    "description": {
                      "urls": []
                    },
                    "url": {
                      "urls": [
                        {
                          "display_url": "scrapfly.io",
                          "expanded_url": "https://scrapfly.io",
                          "url": "https://t.co/1Is3k6KzyM",
                          "indices": [
                            0,
                            23
                          ]
                        }
                      ]
                    }
                  },
                  "fast_followers_count": 0,
                  "favourites_count": 26,
                  "followers_count": 163,
                  "friends_count": 993,
                  "has_custom_timelines": true,
                  "is_translator": false,
                  "listed_count": 2,
                  "location": "Paris",
                  "media_count": 11,
                  "name": "Scrapfly",
                  "normal_followers_count": 163,
                  "pinned_tweet_ids_str": [],
                  "possibly_sensitive": false,
                  "profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
                  "profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
                  "profile_interstitial_type": "",
                  "screen_name": "Scrapfly_dev",
                  "statuses_count": 56,
                  "translator_type": "none",
                  "url": "https://t.co/1Is3k6KzyM",
                  "verified": false,
                  "withheld_in_countries": []
                }
              }
            }
          ]
        }
      },
      "unified_card": {
        "card_fetch_state": "NoCard"
      },
      "edit_control": {
        "edit_tweet_ids": [
          "1587431468141318146"
        ],
        "editable_until_msecs": "1667309929000",
        "is_edit_eligible": true,
        "edits_remaining": "5"
      },
      "is_translatable": false,
      "views": {
        "state": "Enabled"
      },
      "source": "<a href=\"https://www.semrush.com/\" rel=\"nofollow\">Semrush Social Media Tool</a>",
      "legacy": {
        "bookmark_count": 0,
        "bookmarked": false,
        "created_at": "Tue Nov 01 13:08:49 +0000 2022",
        "conversation_id_str": "1587431468141318146",
        "display_text_range": [
          0,
          220
        ],
        "entities": {
          "user_mentions": [],
          "urls": [
            {
              "display_url": "bit.ly/3RNDN6e",
              "expanded_url": "https://bit.ly/3RNDN6e",
              "url": "https://t.co/JLCeZGeDtF",
              "indices": [
                197,
                220
              ]
            }
          ],
          "hashtags": [
            {
              "indices": [
                175,
                187
              ],
              "text": "webscraping"
            },
            {
              "indices": [
                188,
                194
              ],
              "text": "proxy"
            }
          ],
          "symbols": []
        },
        "favorite_count": 0,
        "favorited": false,
        "full_text": "Introduction To Proxies in Web Scraping \ud83d\udccd\n\nWhat do we cover in this article?\ud83d\udc47\n\n\ud83d\udd77 What's a Proxy?\n\ud83d\udd77 IP Protocol Versions - Proxy Protocols\n\ud83d\udd77 Proxy Types\n\ud83d\udd77 Common Proxy Issues\n\n#webscraping #proxy \n\nhttps://t.co/JLCeZGeDtF",
        "is_quote_status": false,
        "lang": "en",
        "possibly_sensitive": false,
        "possibly_sensitive_editable": true,
        "quote_count": 0,
        "reply_count": 1,
        "retweet_count": 2,
        "retweeted": false,
        "user_id_str": "1310623081300402178",
        "id_str": "1587431468141318146"
      },
      "quick_promote_eligibility": {
        "eligibility": "IneligibleUserUnauthorized"
      }
    }
  ]
}

Finding Tweets using Topic Timelines

Now that we know how to scrape individual tweets and Twitter users let's take a look at how to find them.

The obvious way is to use Twitter's search system though to scrape Twitter's search we need to login which is not advisable when web scraping as it can lead to account suspension.

Instead, we can find the latest Tweets following one of many publicly available Twitter topic timelines.

Twitter topic timelines can be found over at topics homepage:

twitter.com/i/topics/picker/home

To scrape it we can use the same background request capturing technique and capture the TopicTimeline endpoint:

Python
ScrapFly
from playwright.sync_api import sync_playwright
from nested_lookup import nested_lookup


def scrape_topic(url: str) -> dict:
    """
    Scrape Twitter Topic timeline for latest public tweets e.g.:
    https://twitter.com/i/topics/853980498816679937
    The list of Twitter topics can be found here: 
    https://twitter.com/i/topics/picker/home
    """
    _xhr_calls = []

    def intercept_response(response):
        """capture all background requests and save them"""
        # we can extract details from background requests
        if response.request.resource_type == "xhr":
            _xhr_calls.append(response)
        return response

    with sync_playwright() as pw:
        browser = pw.chromium.launch()
        context = browser.new_context(viewport={"width": 1920, "height": 1080})
        page = context.new_page()

        # enable background request intercepting:
        page.on("response", intercept_response)
        # go to url and wait for the page to load
        page.goto(url)
        page.wait_for_selector("[data-testid='tweet']")

        # find all tweet background requests:
        topic_calls = [f for f in _xhr_calls if "TopicLandingPage" in f.url]
        tweets = []
        for xhr in topic_calls:
            data = xhr.json()
            xhr_tweets = nested_lookup("tweet_results", data)
            tweets.extend([tweet["result"] for tweet in xhr_tweets])
        return tweets


if __name__ == "__main__":
    # example: "dogs" topic:
    print(scrape_topic("https://twitter.com/i/topics/853980498816679937"))
import json
from typing import Dict

from nested_lookup import nested_lookup
from scrapfly import ScrapeConfig, ScrapflyClient

SCRAPFLY = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"])
BASE_CONFIG = {
    # Twitter.com requires Anti Scraping Protection bypass feature.
    # for more: https://scrapfly.io/docs/scrape-api/anti-scraping-protection
    "asp": True,
    # Twitter.com is javascript-powered web application so it requires
    # headless browsers for scraping
    "render_js": True,
    "country": "CA",  # set prefered country here, for example Canada
}


async def scrape_topic(url: str) -> Dict:
    """
    Scrape Twitter Topic timeline for latest public tweets e.g.:
    https://twitter.com/i/topics/853980498816679937
    The list of Twitter topics can be found here: 
    https://twitter.com/i/topics/picker/home
    """
    result = await scrapfly.async_scrape(ScrapeConfig(
        url, 
        wait_for_selector="[data-testid='tweet']"
        **BASE_CONFIG
    ))
    # capture background requests and extract ones that contain user data
    # and their latest tweets
    _xhr_calls = result.scrape_result["browser_data"]["xhr_call"]
    topic_calls = [f for f in _xhr_calls if "TopicLandingPage" in f["url"]]
    tweets = []
    for xhr in topic_calls:
        if not xhr["response"]:
            continue
        data = json.loads(xhr["response"]["body"])
        xhr_tweets = nested_lookup("tweet_results", data)
        tweets.extend([parse_tweet(tweet["result"]) for tweet in xhr_tweets])
    return tweets

Above just like before we startup a headless browser, navigate to a page and capture all background requests. Then, we find the background request that contains TopicLandingPage in the request url and parse it's response for Tweet data attached to this topic.

Bypass Twitter Blocking with ScrapFly

If we start scraping Twitter at scale we start to quickly run into blocking as Twitter doesn't allow automated requests and will block scrapers IP address after extended use.

To get around this we can use proxies and web scraping APIs such as ScrapFly.

scrapfly middleware
ScrapFly service does the heavy lifting for you!

ScrapFly is essentially a HTTP proxy on steroids which provides web scraping abstraction features like:

And much more. To use ScrapFly with Python we can take advantage of Python SDK:

from scrapfly import ScrapflyClient, ScrapeConfig
scrapfly = ScrapflyClient(key="YOUR SCRAPFLY KEY")

result = scrapfly.scrape(ScrapeConfig(
    "https://twitter.com/Scrapfly_dev",
    # we can enable features like:
    # cloud headless browser use
    render_js=True,  
    # anti scraping protection bypass
    asp=True, 
    # screenshot taking
    screenshots={"all": "fullpage"},
    # proxy country selection
    country="US",
))

For more on using ScrapFly to scrape Twitter see the Full Scraper Code section.

FAQ

To wrap up this Python Twitter scraper let's take a look at some frequently asked questions regarding web scraping Twitter:

Yes, all of the data on Twitter is available publically so it's perfectly legal to scrape. However, note that some tweets can contain copyrighted material like images or videos and scraping them can be illegal (though scraping the URLs are perfectly fine).

How to scrape Twitter without getting blocked?

Twitter is a complex javascript-heavy website and is hostile to web scraping so it's easy to get blocked. To avoid this you can use ScrapFly which provides anti scraping technology bypass and proxy rotation. Alternatively, see our article on how to avoid web scraper blocking.

The legality of scraping Twitter while logged in is a bit of a grey area. Generally, logging in legally binds the user to the website's terms of service which in Twitter's case forbids automated scraping. This allows Twitter to suspend your account or even take legal action. It's best to avoid web scraping Twitter while logged in if possible.

How to reduce bandwidth use and speed up Twitter scraping?

If you're using browser automation tools like Playwright (used in this article) then you can block images and unnecessary resources to save bandwidth and speed up scraping.

Latest Twitter.com Scraper Code
https://github.com/scrapfly/scrapfly-scrapers/

Twitter Scraping Summary

In this tutorial, we've taken a look at how to scrape data from Twitter using Python headless browser's through Playwright or Scrapfly SDK.

To start, we've taken a look at how Twitter works and identified where the data is located. We found that Twitter is using background request to populate Tweet, profile and timeline pages.

To capture and scrape these background request we used the intercept functions and parsed the raw datasets to something more usable using jmespath and nested_lookup libraries.

Finally, to avoid blocking we've taken a look at ScrapFly web scraping API which provides a simple way to scrape Twitter at scale using proxies and anti scraping technology bypassing. Try out ScrapFly for free!

Related Posts

How to scrape Threads by Meta using Python (2023-08 Update)

Guide how to scrape Threads - new social media network by Meta and Instagram - using Python and popular libraries like Playwright and background request capture techniques.

How to Scrape Goat.com for Fashion Apparel Data in Python

Goat.com is a rising storefront for luxury fashion apparel items. It's known for high quality apparel data so in this tutorial we'll take a look how to scrape it using Python.

How to Scrape Fashionphile for Second Hand Fashion Data

In this fashion scrapeguide we'll be taking a look at Fashionphile - another major 2nd hand luxury fashion marketplace. We'll be using Python and hidden web data scraping to grap all of this data in just few lines of code.