How to scrape Threads by Meta using Python (2023-08 Update)
Guide how to scrape Threads - new social media network by Meta and Instagram - using Python and popular libraries like Playwright and background request capture techniques.
With the recent news of Twitter closing its API to new developers, we decided to write a tutorial on how to scrape Twitter.
In this tutorial, we'll be using nothing but Python to retrieve Twitter data such as:
We'll be scraping Twitter without login or any complex tricks using headless browsers and capturing background requests making this a very simple and powerful scraper.
For our headless browser environment, we'll be using Scrapfly SDK with Javascript Rendering feature. Alternatively, for non scraply users we'll also show how to achieve similar results using Playwright.
Twitter is a major announcement hub where people and companies publish their announcements. This is a great opportunity to use Twitter to follow industry trends. For example, stock market or crypto market targets could be scraped to predict the future price of a stock or crypto.
Twitter is also a great source of data for sentiment analysis. You can use Twitter to find out what people think about a certain topic or brand. This is useful for market research, product development, and brand awareness.
So, if we can scrape Twitter data with Python we can have access to this valuable public information for free!
In this tutorial, we'll cover Twitter scraping using Python and Scrapfly-sdk or Playwright.
To parse the scraped Twitter datasets we'll be using Jmespath JSON parsing library which allows to reshape JSON data and nested-lookup which allows to search nested JSON data.
All of these libraries are available for free and can be installed via pip install terminal command:
$ pip install playwright jmespath nested-lookup "scrapfly-sdk[all]"
For an introduction to web scraping with Playwright see this beginner's guide which covers common functionality and an example project.
Before we start scraping, let's take a quick look at how does the Twitter.com website works so we can understand how to scrape it.
To start, Twitter is a javascript web application so scraping it without a headless browser such as Playwright or Scrapfly SDK would be very difficult as we'd have to reverse engineer the entire Twitter API and application process.
Next, Twitter page HTML is dynamic and complex making parsing scraped content very difficult.
So, the best approach to scrape Twitter is to use a headless browser and capture background requests that download the Tweet and user data.
For example, if we take a look at a Twitter profile page in Browser Developer Tools we can see the requests Twitter performs in the background to load the page data:
So, to scrape Twitter we'll be using a headless browser (like Playwright) and capture these background requests to retrieve the data we need!
Recently, Twitter has introduced a temporary login requirement for viewing posts and pages. While this is a temporary measure, it does make scraping Twitter more difficult.
One way to get past this requirement is to scrape Twitter Tweet and Timeline embed widget instead of the website itself.
Let's take a look at how to scrape Twitter embed using Python.
To scrape Twitter Timelines we can use the timeline embed widget. This widget can be found via URL pattern:
https://syndication.twitter.com/srv/timeline-profile/screen-name/USER
# e.g.
https://syndication.twitter.com/srv/timeline-profile/screen-name/scrapfly_dev
To scrape it using python
import json
import httpx
from parsel import Selector
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.54 Safari/537.36",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9",
}
# retrieve embed HTML
with httpx.Client(http2=True, headers=HEADERS) as client:
response = client.get(
"https://syndication.twitter.com/srv/timeline-profile/screen-name/scrapfly_dev"
)
assert response.status_code == 200
sel = Selector(response.text)
# find data cache:
data = json.loads(sel.css("script#__NEXT_DATA__::text").get())
# parse tweet data from data cache JSON:
tweet_data = data["props"]["pageProps"]["timeline"]["entries"]
tweets = [tweet["content"]["tweet"] for tweet in tweet_data]
print(tweets)
import json
from scrapfly import ScrapflyClient, ScrapeConfig
scrapfly = ScrapflyClient("YOUR SCRAPFLY KEY")
result = scrapfly.scrape(
ScrapeConfig("https://syndication.twitter.com/srv/timeline-profile/screen-name/scrapfly_dev")
)
# find data cache:
data = json.loads(result.selector.css("script#__NEXT_DATA__::text").get())
# parse tweet data from data cache JSON:
tweet_data = data["props"]["pageProps"]["timeline"]["entries"]
tweets = [tweet["content"]["tweet"] for tweet in tweet_data]
print(tweets)
In the example above, we're scraping the Twitter-embedded timeline and then parsing hidden web data for all timeline tweet information. For refining the dataset see the Tweet Parsing section.
To scrape Tweets without login we can use the Tweet embed widget. This widget is can be accessed through url pattern:
https://platform.twitter.com/embed/Tweet.html?id=TWEET_ID
# where TWEET_ID is numeric id of the tweet
https://twitter.com/Scrapfly_dev/status/1664267318053179398
# ^^^^^^^^^^^^^^^^^^
https://platform.twitter.com/embed/Tweet.html?id=1664267318053179398
However, that's not the direct URL to the Tweet's data. If we take a look at what the browser does when we visit this page:
So, to scrape this we'll be scraping this URL instead:
https://cdn.syndication.twimg.com/tweet-result?id=1664267318053179398&lang=en
Which is very simply done using Python:
import httpx
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.54 Safari/537.36",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9",
}
# retrieve embed HTML
with httpx.Client(http2=True, headers=HEADERS) as client:
response = client.get(
"https://cdn.syndication.twimg.com/tweet-result?id=1664267318053179398&lang=en",
)
assert response.status_code == 200
data = response.json()
print(data)
import json
from scrapfly import ScrapflyClient, ScrapeConfig
scrapfly = ScrapflyClient("YOUR SCRAPFLY KEY")
result = scrapfly.scrape(
ScrapeConfig("https://cdn.syndication.twimg.com/tweet-result?id=1664267318053179398&lang=en")
)
data = json.loads(result.content)
print(data)
Above we're calling embed tweet widget API directly though Twitter's CDN. By providing Tweet numeric ID we can scrape the entire dataset of the Tweet.
To scrape individual tweet pages we'll be loading the page using a headless browser and capturing the background requests that connect to TweetDetail graphql endpoint.
This background requests returns a JSON response that contains all of the Tweet and user information.
So, to scrape this using Python we can either use Playwright or Scrapfly SDK:
from playwright.sync_api import sync_playwright
from nested_lookup import nested_lookup
def scrape_tweet(url: str) -> dict:
"""
Scrape a single tweet page for Tweet thread e.g.:
https://twitter.com/Scrapfly_dev/status/1667013143904567296
Return parent tweet, reply tweets and recommended tweets
"""
_xhr_calls = []
def intercept_response(response):
"""capture all background requests and save them"""
# we can extract details from background requests
if response.request.resource_type == "xhr":
_xhr_calls.append(response)
return response
with sync_playwright() as pw:
browser = pw.chromium.launch()
context = browser.new_context(viewport={"width": 1920, "height": 1080})
page = context.new_page()
# enable background request intercepting:
page.on("response", intercept_response)
# go to url and wait for the page to load
page.goto(url)
page.wait_for_selector("[data-testid='tweet']")
# find all tweet background requests:
tweet_calls = [f for f in _xhr_calls if "TweetDetail" in f.url]
tweets = []
for xhr in tweet_calls:
data = xhr.json()
xhr_tweets = nested_lookup("tweet_results", data)
tweets.extend([tweet["result"] for tweet in xhr_tweets])
# Now that we have all tweets we can parse them into a thread
# The first tweet is the parent, the rest are replies or suggested tweets
parent = tweets.pop(0)
replies = []
other = []
for tweet in tweets:
if tweet["conversation_id"] == parent["conversation_id"]:
replies.append(tweet)
else:
other.append(tweet)
return {
"tweet": parent,
"replies": replies,
"other": other, # ads, recommended etc
}
if __name__ == "__main__":
print(scrape_tweet("https://twitter.com/Scrapfly_dev/status/1664267318053179398"))
import json
from typing import Dict
from nested_lookup import nested_lookup
from scrapfly import ScrapeConfig, ScrapflyClient
SCRAPFLY = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"])
BASE_CONFIG = {
# Twitter.com requires Anti Scraping Protection bypass feature.
# for more: https://scrapfly.io/docs/scrape-api/anti-scraping-protection
"asp": True,
# Twitter.com is javascript-powered web application so it requires
# headless browsers for scraping
"render_js": True,
"country": "CA", # set prefered country here, for example Canada
}
async def scrape_tweet(url: str) -> Dict:
"""
Scrape a single tweet page for Tweet thread e.g.:
https://twitter.com/Scrapfly_dev/status/1667013143904567296
Return parent tweet, reply tweets and recommended tweets
"""
result = await scrapfly.async_scrape(ScrapeConfig(
url,
wait_for_selector="[data-testid='tweet']"
**BASE_CONFIG
))
# capture background requests and extract ones that request Tweet data
_xhr_calls = result.scrape_result["browser_data"]["xhr_call"]
tweet_call = [f for f in _xhr_calls if "TweetDetail" in f["url"]]
tweets = []
for xhr in tweet_call:
if not xhr["response"]:
continue
data = json.loads(xhr["response"]["body"])
# find tweet_results key recursive as that's where tweet data is located
xhr_tweets = nested_lookup("tweet_results", data)
tweets.extend([parse_tweet(tweet["result"]) for tweet in xhr_tweets])
# Now that we have all tweets we can parse them into a thread
# The first tweet is the parent, the rest are replies or suggested tweets
parent = tweets.pop(0)
replies = []
other = []
for tweet in tweets:
if tweet["conversation_id"] == parent["conversation_id"]:
replies.append(tweet)
else:
other.append(tweet)
return {
"tweet": parent,
"replies": replies,
"other": other, # ads, recommended etc
}
if __name__ == "__main__":
import asyncio
asyncio.run(scrape_tweet("https://twitter.com/Scrapfly_dev/status/1664267318053179398")
{
"tweet": {
"__typename": "Tweet",
"rest_id": "1664267318053179398",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1664267318053179398"
],
"editable_until_msecs": "1685629023000",
"is_edit_eligible": true,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "43",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Thu Jun 01 13:47:03 +0000 2023",
"conversation_id_str": "1664267318053179398",
"display_text_range": [
0,
122
],
"entities": {
"media": [
{
"display_url": "pic.twitter.com/zLjDlxdKee",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1664267318053179398/photo/1",
"id_str": "1664267314160607232",
"indices": [
123,
146
],
"media_url_https": "https://pbs.twimg.com/media/FxiqTffWIAALf7O.png",
"type": "photo",
"url": "https://t.co/zLjDlxdKee",
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
],
"user_mentions": [],
"urls": [
{
"display_url": "scrapfly.io/blog/top-10-we\u2026",
"expanded_url": "https://scrapfly.io/blog/top-10-web-scraping-libraries-in-python/",
"url": "https://t.co/d2iFdAV2LJ",
"indices": [
99,
122
]
}
],
"hashtags": [],
"symbols": []
},
"extended_entities": {
"media": [
{
"display_url": "pic.twitter.com/zLjDlxdKee",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1664267318053179398/photo/1",
"id_str": "1664267314160607232",
"indices": [
123,
146
],
"media_key": "3_1664267314160607232",
"media_url_https": "https://pbs.twimg.com/media/FxiqTffWIAALf7O.png",
"type": "photo",
"url": "https://t.co/zLjDlxdKee",
"ext_media_availability": {
"status": "Available"
},
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
]
},
"favorite_count": 0,
"favorited": false,
"full_text": "A new blog post has been published! \n\nTop 10 Web Scraping Packages for Python \ud83e\udd16\n\nCheckout it out \ud83d\udc47\nhttps://t.co/d2iFdAV2LJ https://t.co/zLjDlxdKee",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 0,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1664267318053179398"
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
"replies": [],
"other": []
}
Here, we loaded the Tweet page using a headless browser and captured all of the background requests. Then, we filtered out the ones that contained the Tweet data and extract that.
One important note here is that we need to wait for the page to load which is indicated by tweets appearing on the page HTML otherwise we'll return our scrape before the background requests have finished.
This resulted in a massive JSON dataset that can be difficult to work with. So, let's take a look at how to reduce it with a bit of JSON parsing next.
The Tweet dataset we scraped contains a lot of complex data so let's reduce it to something more clean and simple using Jmespath JSON parsing library.
For this, we'll be using jmespath's JSON reshaping feature which allows us to rename keys and flatten nested objects:
from typing import Dict
def parse_tweet(data: Dict) -> Dict:
"""Parse Twitter tweet JSON dataset for the most important fields"""
result = jmespath.search(
"""{
created_at: legacy.created_at,
attached_urls: legacy.entities.urls[].expanded_url,
attached_urls2: legacy.entities.url.urls[].expanded_url,
attached_media: legacy.entities.media[].media_url_https,
tagged_users: legacy.entities.user_mentions[].screen_name,
tagged_hashtags: legacy.entities.hashtags[].text,
favorite_count: legacy.favorite_count,
bookmark_count: legacy.bookmark_count,
quote_count: legacy.quote_count,
reply_count: legacy.reply_count,
retweet_count: legacy.retweet_count,
quote_count: legacy.quote_count,
text: legacy.full_text,
is_quote: legacy.is_quote_status,
is_retweet: legacy.retweeted,
language: legacy.lang,
user_id: legacy.user_id_str,
id: legacy.id_str,
conversation_id: legacy.conversation_id_str,
source: source,
views: views.count
}""",
data,
)
result["poll"] = {}
poll_data = jmespath.search("card.legacy.binding_values", data) or []
for poll_entry in poll_data:
key, value = poll_entry["key"], poll_entry["value"]
if "choice" in key:
result["poll"][key] = value["string_value"]
elif "end_datetime" in key:
result["poll"]["end"] = value["string_value"]
elif "last_updated_datetime" in key:
result["poll"]["updated"] = value["string_value"]
elif "counts_are_final" in key:
result["poll"]["ended"] = value["boolean_value"]
elif "duration_minutes" in key:
result["poll"]["duration"] = value["string_value"]
user_data = jmespath.search("core.user_results.result", data)
if user_data:
result["user"] = parse_user(user_data)
return result
Above we're using jmespath to reshape the giant, nested dataset we scraped from Twitter's graphql backend into a flat dictionary containing only the most important fields.
To scrape Twitter profile pages we'll be using the same background request capturing approach just this time we'll be capturing UserBy endpoints which contain the user data in addition to UserTweets endpoint which contains the user's tweets.
We'll be using the same technique we used to scrape Tweets just with a few minor adjustments:
from playwright.sync_api import sync_playwright
from nested_lookup import nested_lookup
def scrape_profile(url: str) -> dict:
"""
Scrapes Twitter user profile page e.g.:
https://twitter.com/scrapfly_dev
returns user data and latest tweets
"""
_xhr_calls = []
def intercept_response(response):
"""capture all background requests and save them"""
# we can extract details from background requests
if response.request.resource_type == "xhr":
_xhr_calls.append(response)
return response
with sync_playwright() as pw:
browser = pw.chromium.launch()
context = browser.new_context(viewport={"width": 1920, "height": 1080})
page = context.new_page()
# enable intercepting for this page
page.on("response", intercept_response)
page.goto(url)
page.wait_for_selector("[data-testid='tweet']")
user_calls = [f for f in _xhr_calls if "UserBy" in f.url]
users = {}
for xhr in user_calls:
data = xhr.json()
user_data = data["data"]["user"]["result"]
users[user_data["legacy"]["screen_name"]] = user_data
tweet_calls = [f for f in _xhr_calls if "UserTweets" in f.url]
tweets = []
for xhr in tweet_calls:
data = xhr.json()
xhr_tweets = nested_lookup("tweet_results", data)
tweets.extend([tweet["result"] for tweet in xhr_tweets])
users[user_data["legacy"]["screen_name"]] = user_data
return {"users": users, "tweets": tweets}
if __name__ == "__main__":
print(scrape_profile("https://twitter.com/Scrapfly_dev"))
import json
from typing import Dict
from nested_lookup import nested_lookup
from scrapfly import ScrapeConfig, ScrapflyClient
SCRAPFLY = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"])
BASE_CONFIG = {
# Twitter.com requires Anti Scraping Protection bypass feature.
# for more: https://scrapfly.io/docs/scrape-api/anti-scraping-protection
"asp": True,
# Twitter.com is javascript-powered web application so it requires
# headless browsers for scraping
"render_js": True,
"country": "CA", # set prefered country here, for example Canada
}
async def scrape_profile(url: str) -> Dict:
"""
Scrapes Twitter user profile page e.g.:
https://twitter.com/scrapfly_dev
returns user data and latest tweets
"""
result = await scrapfly.async_scrape(ScrapeConfig(
url,
wait_for_selector="[data-testid='tweet']"
**BASE_CONFIG
))
# capture background requests and extract ones that contain user data
# and their latest tweets
_xhr_calls = result.scrape_result["browser_data"]["xhr_call"]
user_calls = [f for f in _xhr_calls if "UserBy" in f["url"]]
users = {}
for xhr in user_calls:
data = json.loads(xhr["response"]["body"])
parsed = parse_user(data["data"]["user"]["result"])
users[parsed["screen_name"]] = parsed
tweet_paging_calls = [f for f in _xhr_calls if "UserTweets" in f["url"]]
tweets = []
for xhr in tweet_paging_calls:
data = json.loads(xhr["response"]["body"])
xhr_tweets = nested_lookup("tweet_results", data)
tweets.extend([parse_tweet(tweet["result"]) for tweet in xhr_tweets])
return {
"tweets": tweets,
"users": users,
}
if __name__ == "__main__":
import asyncio
asyncio.run(scrape_profile("https://twitter.com/Scrapfly_dev")
{
"users": {
"Scrapfly_dev": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
},
"business_account": {},
"highlights_info": {
"can_highlight_tweets": true,
"highlighted_tweets": "0"
},
"creator_subscriptions_count": 0
}
},
"tweets": [
{
"__typename": "Tweet",
"rest_id": "1667013143904567296",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1667013143904567296"
],
"editable_until_msecs": "1686285479000",
"is_edit_eligible": true,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "35",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Fri Jun 09 03:37:59 +0000 2023",
"conversation_id_str": "1667013143904567296",
"display_text_range": [
0,
139
],
"entities": {
"media": [
{
"display_url": "pic.twitter.com/mLnjGwMnmD",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1667013143904567296/photo/1",
"id_str": "1667013141392179201",
"indices": [
140,
163
],
"media_url_https": "https://pbs.twimg.com/media/FyJrnrCWwAEpvzT.png",
"type": "photo",
"url": "https://t.co/mLnjGwMnmD",
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
],
"user_mentions": [],
"urls": [
{
"display_url": "scrapfly.io/blog/parsing-d\u2026",
"expanded_url": "https://scrapfly.io/blog/parsing-datetime-strings-with-python-and-dateparser/",
"url": "https://t.co/FeEpxzQ3sK",
"indices": [
116,
139
]
}
],
"hashtags": [],
"symbols": []
},
"extended_entities": {
"media": [
{
"display_url": "pic.twitter.com/mLnjGwMnmD",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1667013143904567296/photo/1",
"id_str": "1667013141392179201",
"indices": [
140,
163
],
"media_key": "3_1667013141392179201",
"media_url_https": "https://pbs.twimg.com/media/FyJrnrCWwAEpvzT.png",
"type": "photo",
"url": "https://t.co/mLnjGwMnmD",
"ext_media_availability": {
"status": "Available"
},
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
]
},
"favorite_count": 0,
"favorited": false,
"full_text": "A new blog post has been published! \n\nHow to Parse Datetime Strings with Python and Dateparser \ud83e\udd16\n\nCheckout it out \ud83d\udc47\nhttps://t.co/FeEpxzQ3sK https://t.co/mLnjGwMnmD",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 0,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1667013143904567296"
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1664267318053179398",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1664267318053179398"
],
"editable_until_msecs": "1685629023000",
"is_edit_eligible": true,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "43",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Thu Jun 01 13:47:03 +0000 2023",
"conversation_id_str": "1664267318053179398",
"display_text_range": [
0,
122
],
"entities": {
"media": [
{
"display_url": "pic.twitter.com/zLjDlxdKee",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1664267318053179398/photo/1",
"id_str": "1664267314160607232",
"indices": [
123,
146
],
"media_url_https": "https://pbs.twimg.com/media/FxiqTffWIAALf7O.png",
"type": "photo",
"url": "https://t.co/zLjDlxdKee",
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
],
"user_mentions": [],
"urls": [
{
"display_url": "scrapfly.io/blog/top-10-we\u2026",
"expanded_url": "https://scrapfly.io/blog/top-10-web-scraping-libraries-in-python/",
"url": "https://t.co/d2iFdAV2LJ",
"indices": [
99,
122
]
}
],
"hashtags": [],
"symbols": []
},
"extended_entities": {
"media": [
{
"display_url": "pic.twitter.com/zLjDlxdKee",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1664267318053179398/photo/1",
"id_str": "1664267314160607232",
"indices": [
123,
146
],
"media_key": "3_1664267314160607232",
"media_url_https": "https://pbs.twimg.com/media/FxiqTffWIAALf7O.png",
"type": "photo",
"url": "https://t.co/zLjDlxdKee",
"ext_media_availability": {
"status": "Available"
},
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
]
},
"favorite_count": 0,
"favorited": false,
"full_text": "A new blog post has been published! \n\nTop 10 Web Scraping Packages for Python \ud83e\udd16\n\nCheckout it out \ud83d\udc47\nhttps://t.co/d2iFdAV2LJ https://t.co/zLjDlxdKee",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 0,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1664267318053179398"
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1658170586508546048",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1658170586508546048"
],
"editable_until_msecs": "1684175449000",
"is_edit_eligible": true,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "45",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Mon May 15 18:00:49 +0000 2023",
"conversation_id_str": "1658170586508546048",
"display_text_range": [
0,
130
],
"entities": {
"media": [
{
"display_url": "pic.twitter.com/1uRcNBdlmv",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1658170586508546048/photo/1",
"id_str": "1658170581689196558",
"indices": [
131,
154
],
"media_url_https": "https://pbs.twimg.com/media/FwMBW9KWYA4B61a.png",
"type": "photo",
"url": "https://t.co/1uRcNBdlmv",
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 1070,
"w": 2048,
"resize": "fit"
},
"medium": {
"h": 627,
"w": 1200,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 1572,
"width": 3008,
"focus_rects": [
{
"x": 101,
"y": 0,
"w": 2807,
"h": 1572
},
{
"x": 718,
"y": 0,
"w": 1572,
"h": 1572
},
{
"x": 815,
"y": 0,
"w": 1379,
"h": 1572
},
{
"x": 1111,
"y": 0,
"w": 786,
"h": 1572
},
{
"x": 0,
"y": 0,
"w": 3008,
"h": 1572
}
]
}
}
],
"user_mentions": [],
"urls": [
{
"display_url": "scrapfly.io/blog/web-scrap\u2026",
"expanded_url": "https://scrapfly.io/blog/web-scraping-microformats/",
"url": "https://t.co/BhEcWS76rY",
"indices": [
107,
130
]
}
],
"hashtags": [],
"symbols": []
},
"extended_entities": {
"media": [
{
"display_url": "pic.twitter.com/1uRcNBdlmv",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1658170586508546048/photo/1",
"id_str": "1658170581689196558",
"indices": [
131,
154
],
"media_key": "3_1658170581689196558",
"media_url_https": "https://pbs.twimg.com/media/FwMBW9KWYA4B61a.png",
"type": "photo",
"url": "https://t.co/1uRcNBdlmv",
"ext_media_availability": {
"status": "Available"
},
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 1070,
"w": 2048,
"resize": "fit"
},
"medium": {
"h": 627,
"w": 1200,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 1572,
"width": 3008,
"focus_rects": [
{
"x": 101,
"y": 0,
"w": 2807,
"h": 1572
},
{
"x": 718,
"y": 0,
"w": 1572,
"h": 1572
},
{
"x": 815,
"y": 0,
"w": 1379,
"h": 1572
},
{
"x": 1111,
"y": 0,
"w": 786,
"h": 1572
},
{
"x": 0,
"y": 0,
"w": 3008,
"h": 1572
}
]
}
}
]
},
"favorite_count": 0,
"favorited": false,
"full_text": "A new blog post has been published! \n\nWeb Scraping Simplified - Scraping Microformats \ud83e\udd16\n\nCheckout it out \ud83d\udc47\nhttps://t.co/BhEcWS76rY https://t.co/1uRcNBdlmv",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 0,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1658170586508546048"
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1653035176631447552",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1653035176631447552"
],
"editable_until_msecs": "1682951072000",
"is_edit_eligible": true,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "58",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Mon May 01 13:54:32 +0000 2023",
"conversation_id_str": "1653035176631447552",
"display_text_range": [
0,
122
],
"entities": {
"media": [
{
"display_url": "pic.twitter.com/dlXAa03pFP",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1653035176631447552/photo/1",
"id_str": "1653035174576246784",
"indices": [
123,
146
],
"media_url_https": "https://pbs.twimg.com/media/FvDCu63X0AAPJex.png",
"type": "photo",
"url": "https://t.co/dlXAa03pFP",
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
],
"user_mentions": [],
"urls": [
{
"display_url": "scrapfly.io/blog/web-scrap\u2026",
"expanded_url": "https://scrapfly.io/blog/web-scraping-with-python-httpx/",
"url": "https://t.co/29fKBIVko3",
"indices": [
99,
122
]
}
],
"hashtags": [],
"symbols": []
},
"extended_entities": {
"media": [
{
"display_url": "pic.twitter.com/dlXAa03pFP",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1653035176631447552/photo/1",
"id_str": "1653035174576246784",
"indices": [
123,
146
],
"media_key": "3_1653035174576246784",
"media_url_https": "https://pbs.twimg.com/media/FvDCu63X0AAPJex.png",
"type": "photo",
"url": "https://t.co/dlXAa03pFP",
"ext_media_availability": {
"status": "Available"
},
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
]
},
"favorite_count": 1,
"favorited": false,
"full_text": "A new blog post has been published! \n\nHow to Web Scrape with HTTPX and Python \ud83e\udd16\n\nCheckout it out \ud83d\udc47\nhttps://t.co/29fKBIVko3 https://t.co/dlXAa03pFP",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 0,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1653035176631447552"
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1651042163885473794",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1651042163885473794"
],
"editable_until_msecs": "1682475901000",
"is_edit_eligible": true,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "55",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Wed Apr 26 01:55:01 +0000 2023",
"conversation_id_str": "1651042163885473794",
"display_text_range": [
0,
139
],
"entities": {
"media": [
{
"display_url": "pic.twitter.com/lrWLxz3R4C",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1651042163885473794/photo/1",
"id_str": "1651042161129840642",
"indices": [
140,
163
],
"media_url_https": "https://pbs.twimg.com/media/FumuGRNWYAIU0QI.png",
"type": "photo",
"url": "https://t.co/lrWLxz3R4C",
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 871,
"w": 1667,
"resize": "fit"
},
"medium": {
"h": 627,
"w": 1200,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 871,
"width": 1667,
"focus_rects": [
{
"x": 56,
"y": 0,
"w": 1555,
"h": 871
},
{
"x": 398,
"y": 0,
"w": 871,
"h": 871
},
{
"x": 451,
"y": 0,
"w": 764,
"h": 871
},
{
"x": 615,
"y": 0,
"w": 436,
"h": 871
},
{
"x": 0,
"y": 0,
"w": 1667,
"h": 871
}
]
}
}
],
"user_mentions": [],
"urls": [
{
"display_url": "scrapfly.io/blog/how-to-sc\u2026",
"expanded_url": "https://scrapfly.io/blog/how-to-scrape-without-getting-blocked-tutorial/",
"url": "https://t.co/tPqNzOy92c",
"indices": [
116,
139
]
}
],
"hashtags": [],
"symbols": []
},
"extended_entities": {
"media": [
{
"display_url": "pic.twitter.com/lrWLxz3R4C",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1651042163885473794/photo/1",
"id_str": "1651042161129840642",
"indices": [
140,
163
],
"media_key": "3_1651042161129840642",
"media_url_https": "https://pbs.twimg.com/media/FumuGRNWYAIU0QI.png",
"type": "photo",
"url": "https://t.co/lrWLxz3R4C",
"ext_media_availability": {
"status": "Available"
},
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 871,
"w": 1667,
"resize": "fit"
},
"medium": {
"h": 627,
"w": 1200,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 871,
"width": 1667,
"focus_rects": [
{
"x": 56,
"y": 0,
"w": 1555,
"h": 871
},
{
"x": 398,
"y": 0,
"w": 871,
"h": 871
},
{
"x": 451,
"y": 0,
"w": 764,
"h": 871
},
{
"x": 615,
"y": 0,
"w": 436,
"h": 871
},
{
"x": 0,
"y": 0,
"w": 1667,
"h": 871
}
]
}
}
]
},
"favorite_count": 0,
"favorited": false,
"full_text": "A new blog post has been published! \n\nHow to Scrape Without Getting Blocked? In-Depth Tutorial \ud83e\udd16\n\nCheckout it out \ud83d\udc47\nhttps://t.co/tPqNzOy92c https://t.co/lrWLxz3R4C",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 0,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1651042163885473794"
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1647938361586221058",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1647938361586221058"
],
"editable_until_msecs": "1681735897000",
"is_edit_eligible": true,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "103",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Mon Apr 17 12:21:37 +0000 2023",
"conversation_id_str": "1647938361586221058",
"display_text_range": [
0,
106
],
"entities": {
"media": [
{
"display_url": "pic.twitter.com/kCohTttrmM",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1647938361586221058/photo/1",
"id_str": "1647938359170416642",
"indices": [
107,
130
],
"media_url_https": "https://pbs.twimg.com/media/Ft6nNN4XwAIuoHL.png",
"type": "photo",
"url": "https://t.co/kCohTttrmM",
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
],
"user_mentions": [],
"urls": [],
"hashtags": [],
"symbols": []
},
"extended_entities": {
"media": [
{
"display_url": "pic.twitter.com/kCohTttrmM",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1647938361586221058/photo/1",
"id_str": "1647938359170416642",
"indices": [
107,
130
],
"media_key": "3_1647938359170416642",
"media_url_https": "https://pbs.twimg.com/media/Ft6nNN4XwAIuoHL.png",
"type": "photo",
"url": "https://t.co/kCohTttrmM",
"ext_media_availability": {
"status": "Available"
},
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
]
},
"favorite_count": 0,
"favorited": false,
"full_text": "A new blog post has been published! \n\nStepping into Footwear Market with Web Scraping \ud83e\udd16\n\nCheckout it out \ud83d\udc47 https://t.co/kCohTttrmM",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 0,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1647938361586221058"
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1646105015344418816",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1646105015344418816"
],
"editable_until_msecs": "1681298793000",
"is_edit_eligible": true,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "59",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Wed Apr 12 10:56:33 +0000 2023",
"conversation_id_str": "1646105015344418816",
"display_text_range": [
0,
131
],
"entities": {
"media": [
{
"display_url": "pic.twitter.com/9WaCUH38NA",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1646105015344418816/photo/1",
"id_str": "1646105012731363330",
"indices": [
132,
155
],
"media_url_https": "https://pbs.twimg.com/media/FtgjyaLXwAIrKBk.png",
"type": "photo",
"url": "https://t.co/9WaCUH38NA",
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
],
"user_mentions": [],
"urls": [
{
"display_url": "Goat.com",
"expanded_url": "http://Goat.com",
"url": "https://t.co/PRRUEhuYJp",
"indices": [
52,
75
]
}
],
"hashtags": [],
"symbols": []
},
"extended_entities": {
"media": [
{
"display_url": "pic.twitter.com/9WaCUH38NA",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1646105015344418816/photo/1",
"id_str": "1646105012731363330",
"indices": [
132,
155
],
"media_key": "3_1646105012731363330",
"media_url_https": "https://pbs.twimg.com/media/FtgjyaLXwAIrKBk.png",
"type": "photo",
"url": "https://t.co/9WaCUH38NA",
"ext_media_availability": {
"status": "Available"
},
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
]
},
"favorite_count": 1,
"favorited": false,
"full_text": "A new blog post has been published! \n\nHow to Scrape https://t.co/PRRUEhuYJp for Fashion Apparel Data in Python \ud83e\udd16\n\nCheckout it out \ud83d\udc47 https://t.co/9WaCUH38NA",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 0,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1646105015344418816"
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1645521299136032769",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1645521299136032769"
],
"editable_until_msecs": "1681159624000",
"is_edit_eligible": true,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "82",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Mon Apr 10 20:17:04 +0000 2023",
"conversation_id_str": "1645521299136032769",
"display_text_range": [
0,
110
],
"entities": {
"media": [
{
"display_url": "pic.twitter.com/7jm5EkjzGx",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1645521299136032769/photo/1",
"id_str": "1645521297001119763",
"indices": [
111,
134
],
"media_url_https": "https://pbs.twimg.com/media/FtYQ5roXgBMOnUH.png",
"type": "photo",
"url": "https://t.co/7jm5EkjzGx",
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
],
"user_mentions": [],
"urls": [],
"hashtags": [],
"symbols": []
},
"extended_entities": {
"media": [
{
"display_url": "pic.twitter.com/7jm5EkjzGx",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1645521299136032769/photo/1",
"id_str": "1645521297001119763",
"indices": [
111,
134
],
"media_key": "3_1645521297001119763",
"media_url_https": "https://pbs.twimg.com/media/FtYQ5roXgBMOnUH.png",
"type": "photo",
"url": "https://t.co/7jm5EkjzGx",
"ext_media_availability": {
"status": "Available"
},
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
]
},
"favorite_count": 1,
"favorited": false,
"full_text": "A new blog post has been published! \n\nHow to Scrape Sitemaps to Discover Scraping Targets \ud83e\udd16\n\nCheckout it out \ud83d\udc47 https://t.co/7jm5EkjzGx",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 0,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1645521299136032769"
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1645490142763786260",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1645490142763786260"
],
"editable_until_msecs": "1681152196000",
"is_edit_eligible": true,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "62",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://zapier.com/\" rel=\"nofollow\">Zapier.com</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Mon Apr 10 18:13:16 +0000 2023",
"conversation_id_str": "1645490142763786260",
"display_text_range": [
0,
114
],
"entities": {
"media": [
{
"display_url": "pic.twitter.com/8KqHNTH8Ar",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1645490142763786260/photo/1",
"id_str": "1645490139924164608",
"indices": [
115,
138
],
"media_url_https": "https://pbs.twimg.com/media/FtX0kGeWcAAH-kz.png",
"type": "photo",
"url": "https://t.co/8KqHNTH8Ar",
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
],
"user_mentions": [],
"urls": [],
"hashtags": [],
"symbols": []
},
"extended_entities": {
"media": [
{
"display_url": "pic.twitter.com/8KqHNTH8Ar",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1645490142763786260/photo/1",
"id_str": "1645490139924164608",
"indices": [
115,
138
],
"media_key": "3_1645490139924164608",
"media_url_https": "https://pbs.twimg.com/media/FtX0kGeWcAAH-kz.png",
"type": "photo",
"url": "https://t.co/8KqHNTH8Ar",
"ext_media_availability": {
"status": "Available"
},
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 416,
"w": 796,
"resize": "fit"
},
"medium": {
"h": 416,
"w": 796,
"resize": "fit"
},
"small": {
"h": 355,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 416,
"width": 796,
"focus_rects": [
{
"x": 27,
"y": 0,
"w": 743,
"h": 416
},
{
"x": 190,
"y": 0,
"w": 416,
"h": 416
},
{
"x": 216,
"y": 0,
"w": 365,
"h": 416
},
{
"x": 294,
"y": 0,
"w": 208,
"h": 416
},
{
"x": 0,
"y": 0,
"w": 796,
"h": 416
}
]
}
}
]
},
"favorite_count": 0,
"favorited": false,
"full_text": "A new blog post has been published! \n\nHow to Scrape Fashionphile for Second Hand Fashion Data \ud83e\udd16\n\nCheckout it out \ud83d\udc47 https://t.co/8KqHNTH8Ar",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 0,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1645490142763786260"
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1628001040154128390",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"card": {
"rest_id": "https://t.co/x6EHAZFElO",
"legacy": {
"binding_values": [
{
"key": "photo_image_full_size_large",
"value": {
"image_value": {
"height": 419,
"width": 800,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=800x419"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image",
"value": {
"image_value": {
"height": 146,
"width": 280,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=280x150"
},
"type": "IMAGE"
}
},
{
"key": "description",
"value": {
"string_value": "Intro to using Python and JSONPath library and a query language for parsing JSON datasets.",
"type": "STRING"
}
},
{
"key": "domain",
"value": {
"string_value": "scrapfly.io",
"type": "STRING"
}
},
{
"key": "thumbnail_image_large",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=600x600"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image_small",
"value": {
"image_value": {
"height": 202,
"width": 386,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=386x202"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_original",
"value": {
"image_value": {
"height": 1572,
"width": 3008,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=orig"
},
"type": "IMAGE"
}
},
{
"key": "site",
"value": {
"scribe_key": "publisher_id",
"type": "USER",
"user_value": {
"id_str": "1310623081300402178",
"path": []
}
}
},
{
"key": "photo_image_full_size_small",
"value": {
"image_value": {
"height": 202,
"width": 386,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=386x202"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image_large",
"value": {
"image_value": {
"height": 419,
"width": 800,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=800x419"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_small",
"value": {
"image_value": {
"height": 75,
"width": 144,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=144x144"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_x_large",
"value": {
"image_value": {
"height": 1070,
"width": 2048,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "photo_image_full_size_original",
"value": {
"image_value": {
"height": 1572,
"width": 3008,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=orig"
},
"type": "IMAGE"
}
},
{
"key": "vanity_url",
"value": {
"scribe_key": "vanity_url",
"string_value": "scrapfly.io",
"type": "STRING"
}
},
{
"key": "photo_image_full_size",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=600x314"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 168,
"red": 77
},
"percentage": 98.78
},
{
"rgb": {
"blue": 249,
"green": 219,
"red": 182
},
"percentage": 1.22
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "title",
"value": {
"string_value": "Quick Intro to Parsing JSON with JSONPath in Python",
"type": "STRING"
}
},
{
"key": "summary_photo_image_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 168,
"red": 77
},
"percentage": 98.78
},
{
"rgb": {
"blue": 249,
"green": 219,
"red": 182
},
"percentage": 1.22
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "summary_photo_image_x_large",
"value": {
"image_value": {
"height": 1070,
"width": 2048,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=600x314"
},
"type": "IMAGE"
}
},
{
"key": "photo_image_full_size_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 168,
"red": 77
},
"percentage": 98.78
},
{
"rgb": {
"blue": 249,
"green": 219,
"red": 182
},
"percentage": 1.22
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "photo_image_full_size_x_large",
"value": {
"image_value": {
"height": 1070,
"width": 2048,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "card_url",
"value": {
"scribe_key": "card_url",
"string_value": "https://t.co/x6EHAZFElO",
"type": "STRING"
}
},
{
"key": "summary_photo_image_original",
"value": {
"image_value": {
"height": 1572,
"width": 3008,
"url": "https://pbs.twimg.com/card_img/1667826278966198274/UAPBJYEr?format=jpg&name=orig"
},
"type": "IMAGE"
}
}
],
"card_platform": {
"platform": {
"audience": {
"name": "production"
},
"device": {
"name": "Swift",
"version": "12"
}
}
},
"name": "summary_large_image",
"url": "https://t.co/x6EHAZFElO",
"user_refs_results": [
{
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
]
}
},
"unified_card": {
"card_fetch_state": "NoCard"
},
"edit_control": {
"edit_tweet_ids": [
"1628001040154128390"
],
"editable_until_msecs": "1676982469000",
"is_edit_eligible": false,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "64",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Tue Feb 21 11:57:49 +0000 2023",
"conversation_id_str": "1628001035011915778",
"display_text_range": [
0,
99
],
"entities": {
"user_mentions": [],
"urls": [
{
"display_url": "scrapfly.io/blog/parse-jso\u2026",
"expanded_url": "https://scrapfly.io/blog/parse-json-jsonpath-python",
"url": "https://t.co/x6EHAZFElO",
"indices": [
76,
99
]
}
],
"hashtags": [
{
"indices": [
56,
68
],
"text": "webScraping"
}
],
"symbols": []
},
"favorite_count": 0,
"favorited": false,
"full_text": "For more see our full intro and how JSONPath is used in #webScraping here:\n\nhttps://t.co/x6EHAZFElO",
"in_reply_to_screen_name": "Scrapfly_dev",
"in_reply_to_status_id_str": "1628001037750784001",
"in_reply_to_user_id_str": "1310623081300402178",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 0,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1628001040154128390",
"self_thread": {
"id_str": "1628001035011915778"
}
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1628001037750784001",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1628001037750784001"
],
"editable_until_msecs": "1676982468000",
"is_edit_eligible": false,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "92",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Tue Feb 21 11:57:48 +0000 2023",
"conversation_id_str": "1628001035011915778",
"display_text_range": [
0,
275
],
"entities": {
"user_mentions": [],
"urls": [],
"hashtags": [
{
"indices": [
24,
30
],
"text": "XPath"
}
],
"symbols": []
},
"favorite_count": 0,
"favorited": false,
"full_text": "JSONPath is inspired by #XPath - a popular HTML parsing path language so the use is very similar.\n\nYou can easily query complex JSON datasets with short, recursive queries like `$..product[price>20]` will select all products with price>20 found ANYWHERE in the dataset.",
"in_reply_to_screen_name": "Scrapfly_dev",
"in_reply_to_status_id_str": "1628001035011915778",
"in_reply_to_user_id_str": "1310623081300402178",
"is_quote_status": false,
"lang": "en",
"quote_count": 0,
"reply_count": 1,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1628001037750784001",
"self_thread": {
"id_str": "1628001035011915778"
}
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1628001035011915778",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1628001035011915778"
],
"editable_until_msecs": "1676982468000",
"is_edit_eligible": false,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "148",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Tue Feb 21 11:57:48 +0000 2023",
"conversation_id_str": "1628001035011915778",
"display_text_range": [
0,
172
],
"entities": {
"user_mentions": [],
"urls": [],
"hashtags": [
{
"indices": [
0,
9
],
"text": "jsonpath"
},
{
"indices": [
100,
107
],
"text": "python"
},
{
"indices": [
109,
120
],
"text": "javascript"
},
{
"indices": [
122,
127
],
"text": "ruby"
},
{
"indices": [
129,
131
],
"text": "r"
},
{
"indices": [
136,
139
],
"text": "go"
},
{
"indices": [
159,
171
],
"text": "webScraping"
}
],
"symbols": []
},
"favorite_count": 0,
"favorited": false,
"full_text": "#jsonpath is a path language for parsing JSON datasets and it's pretty awesome! \nIt's available in #python, #javascript, #ruby, #r and #go and is perfect for #webScraping\ud83e\uddf5",
"is_quote_status": false,
"lang": "en",
"quote_count": 0,
"reply_count": 1,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1628001035011915778",
"self_thread": {
"id_str": "1628001035011915778"
}
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1625776258146795520",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"card": {
"rest_id": "https://t.co/mJjtOXGh57",
"legacy": {
"binding_values": [
{
"key": "photo_image_full_size_large",
"value": {
"image_value": {
"height": 419,
"width": 800,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=800x419"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image",
"value": {
"image_value": {
"height": 146,
"width": 280,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=280x150"
},
"type": "IMAGE"
}
},
{
"key": "description",
"value": {
"string_value": "The visible HTML doesn't always represent the whole dataset available on the page. In this article, we'll be taking a look at scraping of hidden web data. What is it and how can we scrape it using...",
"type": "STRING"
}
},
{
"key": "domain",
"value": {
"string_value": "scrapfly.io",
"type": "STRING"
}
},
{
"key": "thumbnail_image_large",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=600x600"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image_small",
"value": {
"image_value": {
"height": 202,
"width": 386,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=386x202"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_original",
"value": {
"image_value": {
"height": 1572,
"width": 3008,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=orig"
},
"type": "IMAGE"
}
},
{
"key": "site",
"value": {
"scribe_key": "publisher_id",
"type": "USER",
"user_value": {
"id_str": "1310623081300402178",
"path": []
}
}
},
{
"key": "photo_image_full_size_small",
"value": {
"image_value": {
"height": 202,
"width": 386,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=386x202"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image_large",
"value": {
"image_value": {
"height": 419,
"width": 800,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=800x419"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_small",
"value": {
"image_value": {
"height": 75,
"width": 144,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=144x144"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_x_large",
"value": {
"image_value": {
"height": 1070,
"width": 2048,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "photo_image_full_size_original",
"value": {
"image_value": {
"height": 1572,
"width": 3008,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=orig"
},
"type": "IMAGE"
}
},
{
"key": "vanity_url",
"value": {
"scribe_key": "vanity_url",
"string_value": "scrapfly.io",
"type": "STRING"
}
},
{
"key": "photo_image_full_size",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=600x314"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 168,
"red": 78
},
"percentage": 98.74
},
{
"rgb": {
"blue": 248,
"green": 216,
"red": 176
},
"percentage": 1.26
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "title",
"value": {
"string_value": "How to Scrape Hidden Web Data",
"type": "STRING"
}
},
{
"key": "summary_photo_image_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 168,
"red": 78
},
"percentage": 98.74
},
{
"rgb": {
"blue": 248,
"green": 216,
"red": 176
},
"percentage": 1.26
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "summary_photo_image_x_large",
"value": {
"image_value": {
"height": 1070,
"width": 2048,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=600x314"
},
"type": "IMAGE"
}
},
{
"key": "photo_image_full_size_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 168,
"red": 78
},
"percentage": 98.74
},
{
"rgb": {
"blue": 248,
"green": 216,
"red": 176
},
"percentage": 1.26
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "photo_image_full_size_x_large",
"value": {
"image_value": {
"height": 1070,
"width": 2048,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "card_url",
"value": {
"scribe_key": "card_url",
"string_value": "https://t.co/mJjtOXGh57",
"type": "STRING"
}
},
{
"key": "summary_photo_image_original",
"value": {
"image_value": {
"height": 1572,
"width": 3008,
"url": "https://pbs.twimg.com/card_img/1668147669716414465/E29eKcKa?format=png&name=orig"
},
"type": "IMAGE"
}
}
],
"card_platform": {
"platform": {
"audience": {
"name": "production"
},
"device": {
"name": "Swift",
"version": "12"
}
}
},
"name": "summary_large_image",
"url": "https://t.co/mJjtOXGh57",
"user_refs_results": [
{
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
]
}
},
"unified_card": {
"card_fetch_state": "NoCard"
},
"edit_control": {
"edit_tweet_ids": [
"1625776258146795520"
],
"editable_until_msecs": "1676452040000",
"is_edit_eligible": false,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "60",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Wed Feb 15 08:37:20 +0000 2023",
"conversation_id_str": "1625775862946893824",
"display_text_range": [
0,
248
],
"entities": {
"user_mentions": [],
"urls": [
{
"display_url": "scrapfly.io/blog/how-to-sc\u2026",
"expanded_url": "https://scrapfly.io/blog/how-to-scrape-hidden-web-data/",
"url": "https://t.co/mJjtOXGh57",
"indices": [
225,
248
]
}
],
"hashtags": [
{
"indices": [
76,
85
],
"text": "selenium"
},
{
"indices": [
87,
97
],
"text": "puppeteer"
},
{
"indices": [
101,
112
],
"text": "playwright"
},
{
"indices": [
190,
197
],
"text": "python"
}
],
"symbols": []
},
"favorite_count": 1,
"favorited": false,
"full_text": "So, to scrape hidden web data we don't need to use a real web browser using #selenium, #puppeteer or #playwright. We can grab the javacript variable dataset by parsing the script with basic #python!\n\nSee our full guide here: https://t.co/mJjtOXGh57",
"in_reply_to_screen_name": "Scrapfly_dev",
"in_reply_to_status_id_str": "1625776006555635714",
"in_reply_to_user_id_str": "1310623081300402178",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 0,
"retweet_count": 1,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1625776258146795520",
"self_thread": {
"id_str": "1625775862946893824"
}
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1625776006555635714",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1625776006555635714"
],
"editable_until_msecs": "1676451980000",
"is_edit_eligible": false,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "73",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Wed Feb 15 08:36:20 +0000 2023",
"conversation_id_str": "1625775862946893824",
"display_text_range": [
0,
88
],
"entities": {
"media": [
{
"display_url": "pic.twitter.com/Ew4p5B9CEJ",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1625776006555635714/photo/1",
"id_str": "1625775987895177216",
"indices": [
89,
112
],
"media_url_https": "https://pbs.twimg.com/media/Fo_qpbLaEAARHcN.jpg",
"type": "photo",
"url": "https://t.co/Ew4p5B9CEJ",
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 567,
"w": 1000,
"resize": "fit"
},
"medium": {
"h": 567,
"w": 1000,
"resize": "fit"
},
"small": {
"h": 386,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 567,
"width": 1000,
"focus_rects": [
{
"x": 0,
"y": 0,
"w": 1000,
"h": 560
},
{
"x": 0,
"y": 0,
"w": 567,
"h": 567
},
{
"x": 27,
"y": 0,
"w": 497,
"h": 567
},
{
"x": 133,
"y": 0,
"w": 284,
"h": 567
},
{
"x": 0,
"y": 0,
"w": 1000,
"h": 567
}
]
}
}
],
"user_mentions": [],
"urls": [],
"hashtags": [],
"symbols": []
},
"extended_entities": {
"media": [
{
"display_url": "pic.twitter.com/Ew4p5B9CEJ",
"expanded_url": "https://twitter.com/Scrapfly_dev/status/1625776006555635714/photo/1",
"id_str": "1625775987895177216",
"indices": [
89,
112
],
"media_key": "3_1625775987895177216",
"media_url_https": "https://pbs.twimg.com/media/Fo_qpbLaEAARHcN.jpg",
"type": "photo",
"url": "https://t.co/Ew4p5B9CEJ",
"ext_media_availability": {
"status": "Available"
},
"features": {
"large": {
"faces": []
},
"medium": {
"faces": []
},
"small": {
"faces": []
},
"orig": {
"faces": []
}
},
"sizes": {
"large": {
"h": 567,
"w": 1000,
"resize": "fit"
},
"medium": {
"h": 567,
"w": 1000,
"resize": "fit"
},
"small": {
"h": 386,
"w": 680,
"resize": "fit"
},
"thumb": {
"h": 150,
"w": 150,
"resize": "crop"
}
},
"original_info": {
"height": 567,
"width": 1000,
"focus_rects": [
{
"x": 0,
"y": 0,
"w": 1000,
"h": 560
},
{
"x": 0,
"y": 0,
"w": 567,
"h": 567
},
{
"x": 27,
"y": 0,
"w": 497,
"h": 567
},
{
"x": 133,
"y": 0,
"w": 284,
"h": 567
},
{
"x": 0,
"y": 0,
"w": 1000,
"h": 567
}
]
}
}
]
},
"favorite_count": 0,
"favorited": false,
"full_text": "For example, here's how Zillow stores it's real estate property data in the page source: https://t.co/Ew4p5B9CEJ",
"in_reply_to_screen_name": "Scrapfly_dev",
"in_reply_to_status_id_str": "1625775929338515456",
"in_reply_to_user_id_str": "1310623081300402178",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 1,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1625776006555635714",
"self_thread": {
"id_str": "1625775862946893824"
}
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1625775929338515456",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1625775929338515456"
],
"editable_until_msecs": "1676451961000",
"is_edit_eligible": false,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "52",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Wed Feb 15 08:36:01 +0000 2023",
"conversation_id_str": "1625775862946893824",
"display_text_range": [
0,
270
],
"entities": {
"user_mentions": [],
"urls": [],
"hashtags": [
{
"indices": [
242,
250
],
"text": "reactjs"
},
{
"indices": [
251,
257
],
"text": "vuejs"
},
{
"indices": [
262,
269
],
"text": "nextjs"
}
],
"symbols": []
},
"favorite_count": 0,
"favorited": false,
"full_text": "Modern javascript-powered websites often use js variables to store page data. \nThis means scrapers without web browser capabilities cannot see the data in the HTML. Most notably this technique is used by popular javascript frameworks such as #reactjs #vuejs and #nextjs.",
"in_reply_to_screen_name": "Scrapfly_dev",
"in_reply_to_status_id_str": "1625775862946893824",
"in_reply_to_user_id_str": "1310623081300402178",
"is_quote_status": false,
"lang": "en",
"quote_count": 0,
"reply_count": 1,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1625775929338515456",
"self_thread": {
"id_str": "1625775862946893824"
}
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1625775862946893824",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"edit_control": {
"edit_tweet_ids": [
"1625775862946893824"
],
"editable_until_msecs": "1676451945000",
"is_edit_eligible": false,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "127",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Wed Feb 15 08:35:45 +0000 2023",
"conversation_id_str": "1625775862946893824",
"display_text_range": [
0,
78
],
"entities": {
"user_mentions": [],
"urls": [],
"hashtags": [
{
"indices": [
54,
66
],
"text": "webScraping"
}
],
"symbols": []
},
"favorite_count": 1,
"favorited": false,
"full_text": "Web scraping hidden web data is becoming the number 1 #webScraping technique \ud83e\uddf5",
"is_quote_status": false,
"lang": "en",
"quote_count": 0,
"reply_count": 1,
"retweet_count": 1,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1625775862946893824",
"self_thread": {
"id_str": "1625775862946893824"
}
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1625137012776001538",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"card": {
"rest_id": "https://t.co/2kiOGlfKPY",
"legacy": {
"binding_values": [
{
"key": "photo_image_full_size_large",
"value": {
"image_value": {
"height": 416,
"width": 794,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=800x419"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image",
"value": {
"image_value": {
"height": 146,
"width": 280,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=280x150"
},
"type": "IMAGE"
}
},
{
"key": "description",
"value": {
"string_value": "With the news of Twitter dropping free API access we're taking a look at web scraping Twitter using Python for free. In this tutorial we'll cover two methods: using Playwright and Twitter's hidden...",
"type": "STRING"
}
},
{
"key": "domain",
"value": {
"string_value": "scrapfly.io",
"type": "STRING"
}
},
{
"key": "thumbnail_image_large",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=600x600"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image_small",
"value": {
"image_value": {
"height": 202,
"width": 386,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=386x202"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_original",
"value": {
"image_value": {
"height": 416,
"width": 796,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=orig"
},
"type": "IMAGE"
}
},
{
"key": "site",
"value": {
"scribe_key": "publisher_id",
"type": "USER",
"user_value": {
"id_str": "1310623081300402178",
"path": []
}
}
},
{
"key": "photo_image_full_size_small",
"value": {
"image_value": {
"height": 202,
"width": 386,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=386x202"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image_large",
"value": {
"image_value": {
"height": 416,
"width": 794,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=800x419"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_small",
"value": {
"image_value": {
"height": 75,
"width": 144,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=144x144"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_x_large",
"value": {
"image_value": {
"height": 416,
"width": 796,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "photo_image_full_size_original",
"value": {
"image_value": {
"height": 416,
"width": 796,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=orig"
},
"type": "IMAGE"
}
},
{
"key": "vanity_url",
"value": {
"scribe_key": "vanity_url",
"string_value": "scrapfly.io",
"type": "STRING"
}
},
{
"key": "photo_image_full_size",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=600x314"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 168,
"red": 78
},
"percentage": 95.42
},
{
"rgb": {
"blue": 253,
"green": 244,
"red": 232
},
"percentage": 4.58
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "title",
"value": {
"string_value": "How to Scrape Twitter with Python",
"type": "STRING"
}
},
{
"key": "summary_photo_image_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 168,
"red": 78
},
"percentage": 95.42
},
{
"rgb": {
"blue": 253,
"green": 244,
"red": 232
},
"percentage": 4.58
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "summary_photo_image_x_large",
"value": {
"image_value": {
"height": 416,
"width": 796,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=600x314"
},
"type": "IMAGE"
}
},
{
"key": "photo_image_full_size_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 168,
"red": 78
},
"percentage": 95.42
},
{
"rgb": {
"blue": 253,
"green": 244,
"red": 232
},
"percentage": 4.58
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "photo_image_full_size_x_large",
"value": {
"image_value": {
"height": 416,
"width": 796,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "card_url",
"value": {
"scribe_key": "card_url",
"string_value": "https://t.co/2kiOGlfKPY",
"type": "STRING"
}
},
{
"key": "summary_photo_image_original",
"value": {
"image_value": {
"height": 416,
"width": 796,
"url": "https://pbs.twimg.com/card_img/1668234710194028544/ez4081MF?format=png&name=orig"
},
"type": "IMAGE"
}
}
],
"card_platform": {
"platform": {
"audience": {
"name": "production"
},
"device": {
"name": "Swift",
"version": "12"
}
}
},
"name": "summary_large_image",
"url": "https://t.co/2kiOGlfKPY",
"user_refs_results": [
{
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
]
}
},
"unified_card": {
"card_fetch_state": "NoCard"
},
"edit_control": {
"edit_tweet_ids": [
"1625137012776001538"
],
"editable_until_msecs": "1676299632000",
"is_edit_eligible": true,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"count": "298",
"state": "EnabledWithCount"
},
"source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
"legacy": {
"bookmark_count": 4,
"bookmarked": false,
"created_at": "Mon Feb 13 14:17:12 +0000 2023",
"conversation_id_str": "1625137012776001538",
"display_text_range": [
0,
266
],
"entities": {
"user_mentions": [],
"urls": [
{
"display_url": "scrapfly.io/blog/how-to-sc\u2026",
"expanded_url": "https://scrapfly.io/blog/how-to-scrape-twitter/",
"url": "https://t.co/2kiOGlfKPY",
"indices": [
229,
252
]
}
],
"hashtags": [
{
"indices": [
33,
41
],
"text": "Twitter"
},
{
"indices": [
57,
64
],
"text": "Python"
},
{
"indices": [
254,
266
],
"text": "webscraping"
}
],
"symbols": []
},
"favorite_count": 2,
"favorited": false,
"full_text": "Did you know that you can scrape #Twitter for free using #Python? \nTwitter is disabling free API access but that shouldn't stop you from getting that public data for your projects!\n\nWe cover 2 ways to scrape Twitter with Python:\nhttps://t.co/2kiOGlfKPY\n\n#webscraping",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 0,
"retweet_count": 1,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1625137012776001538"
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1592574652445954049",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"card": {
"rest_id": "https://t.co/4LlOIGwawB",
"legacy": {
"binding_values": [
{
"key": "photo_image_full_size_large",
"value": {
"image_value": {
"height": 419,
"width": 800,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=800x419"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image",
"value": {
"image_value": {
"height": 146,
"width": 280,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=280x150"
},
"type": "IMAGE"
}
},
{
"key": "description",
"value": {
"string_value": "Residential proxies are the most popular type of proxies used in web scraping. What makes a good residential proxy and what providers are the best?",
"type": "STRING"
}
},
{
"key": "domain",
"value": {
"string_value": "scrapfly.io",
"type": "STRING"
}
},
{
"key": "thumbnail_image_large",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=600x600"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image_small",
"value": {
"image_value": {
"height": 202,
"width": 386,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=386x202"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_original",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=orig"
},
"type": "IMAGE"
}
},
{
"key": "site",
"value": {
"scribe_key": "publisher_id",
"type": "USER",
"user_value": {
"id_str": "1310623081300402178",
"path": []
}
}
},
{
"key": "photo_image_full_size_small",
"value": {
"image_value": {
"height": 202,
"width": 386,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=386x202"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image_large",
"value": {
"image_value": {
"height": 419,
"width": 800,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=800x419"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_small",
"value": {
"image_value": {
"height": 75,
"width": 144,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=144x144"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_x_large",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "photo_image_full_size_original",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=orig"
},
"type": "IMAGE"
}
},
{
"key": "vanity_url",
"value": {
"scribe_key": "vanity_url",
"string_value": "scrapfly.io",
"type": "STRING"
}
},
{
"key": "photo_image_full_size",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=600x314"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 171,
"red": 86
},
"percentage": 98.26
},
{
"rgb": {
"blue": 247,
"green": 212,
"red": 170
},
"percentage": 1.74
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "title",
"value": {
"string_value": "Top 5 Residential Proxy Providers for Web Scraping",
"type": "STRING"
}
},
{
"key": "summary_photo_image_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 171,
"red": 86
},
"percentage": 98.26
},
{
"rgb": {
"blue": 247,
"green": 212,
"red": 170
},
"percentage": 1.74
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "summary_photo_image_x_large",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=600x314"
},
"type": "IMAGE"
}
},
{
"key": "photo_image_full_size_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 171,
"red": 86
},
"percentage": 98.26
},
{
"rgb": {
"blue": 247,
"green": 212,
"red": 170
},
"percentage": 1.74
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "photo_image_full_size_x_large",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "card_url",
"value": {
"scribe_key": "card_url",
"string_value": "https://t.co/4LlOIGwawB",
"type": "STRING"
}
},
{
"key": "summary_photo_image_original",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1667046788652740611/90KgMXS_?format=png&name=orig"
},
"type": "IMAGE"
}
}
],
"card_platform": {
"platform": {
"audience": {
"name": "production"
},
"device": {
"name": "Swift",
"version": "12"
}
}
},
"name": "summary_large_image",
"url": "https://t.co/4LlOIGwawB",
"user_refs_results": [
{
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
]
}
},
"unified_card": {
"card_fetch_state": "NoCard"
},
"edit_control": {
"edit_tweet_ids": [
"1592574652445954049"
],
"editable_until_msecs": "1668536160000",
"is_edit_eligible": true,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"state": "Enabled"
},
"source": "<a href=\"https://www.semrush.com/\" rel=\"nofollow\">Semrush Social Media Tool</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Tue Nov 15 17:46:00 +0000 2022",
"conversation_id_str": "1592574652445954049",
"display_text_range": [
0,
111
],
"entities": {
"user_mentions": [],
"urls": [
{
"display_url": "bit.ly/3yrRild",
"expanded_url": "https://bit.ly/3yrRild",
"url": "https://t.co/4LlOIGwawB",
"indices": [
88,
111
]
}
],
"hashtags": [
{
"indices": [
56,
68
],
"text": "webscraping"
},
{
"indices": [
69,
86
],
"text": "residentialproxy"
}
],
"symbols": []
},
"favorite_count": 0,
"favorited": false,
"full_text": "Top 5 Residential Proxy Providers for Web Scraping \ud83c\udf10 \ud83e\udd16\n\n#webscraping #residentialproxy\n\nhttps://t.co/4LlOIGwawB",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 0,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1592574652445954049"
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1589966079513673728",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"card": {
"rest_id": "https://t.co/3ooXfi4xpx",
"legacy": {
"binding_values": [
{
"key": "photo_image_full_size_large",
"value": {
"image_value": {
"height": 419,
"width": 800,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=800x419"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image",
"value": {
"image_value": {
"height": 146,
"width": 280,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=280x150"
},
"type": "IMAGE"
}
},
{
"key": "description",
"value": {
"string_value": "Analysis and comparison of some of the most popular proxy providers. What makes a good proxy providers? What features and dangers to look out for?",
"type": "STRING"
}
},
{
"key": "domain",
"value": {
"string_value": "scrapfly.io",
"type": "STRING"
}
},
{
"key": "thumbnail_image_large",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=600x600"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image_small",
"value": {
"image_value": {
"height": 202,
"width": 386,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=386x202"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_original",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=orig"
},
"type": "IMAGE"
}
},
{
"key": "site",
"value": {
"scribe_key": "publisher_id",
"type": "USER",
"user_value": {
"id_str": "1310623081300402178",
"path": []
}
}
},
{
"key": "photo_image_full_size_small",
"value": {
"image_value": {
"height": 202,
"width": 386,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=386x202"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image_large",
"value": {
"image_value": {
"height": 419,
"width": 800,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=800x419"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_small",
"value": {
"image_value": {
"height": 75,
"width": 144,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=144x144"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_x_large",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "photo_image_full_size_original",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=orig"
},
"type": "IMAGE"
}
},
{
"key": "vanity_url",
"value": {
"scribe_key": "vanity_url",
"string_value": "scrapfly.io",
"type": "STRING"
}
},
{
"key": "photo_image_full_size",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=600x314"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 171,
"red": 86
},
"percentage": 98.64
},
{
"rgb": {
"blue": 247,
"green": 212,
"red": 170
},
"percentage": 1.36
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "title",
"value": {
"string_value": "Best Proxy Providers for Web Scraping (2023 Update)",
"type": "STRING"
}
},
{
"key": "summary_photo_image_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 171,
"red": 86
},
"percentage": 98.64
},
{
"rgb": {
"blue": 247,
"green": 212,
"red": 170
},
"percentage": 1.36
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "summary_photo_image_x_large",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=600x314"
},
"type": "IMAGE"
}
},
{
"key": "photo_image_full_size_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 171,
"red": 86
},
"percentage": 98.64
},
{
"rgb": {
"blue": 247,
"green": 212,
"red": 170
},
"percentage": 1.36
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "photo_image_full_size_x_large",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "card_url",
"value": {
"scribe_key": "card_url",
"string_value": "https://t.co/3ooXfi4xpx",
"type": "STRING"
}
},
{
"key": "summary_photo_image_original",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1667154696505630720/X-P7lKqN?format=png&name=orig"
},
"type": "IMAGE"
}
}
],
"card_platform": {
"platform": {
"audience": {
"name": "production"
},
"device": {
"name": "Swift",
"version": "12"
}
}
},
"name": "summary_large_image",
"url": "https://t.co/3ooXfi4xpx",
"user_refs_results": [
{
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
]
}
},
"unified_card": {
"card_fetch_state": "NoCard"
},
"edit_control": {
"edit_tweet_ids": [
"1589966079513673728"
],
"editable_until_msecs": "1667914228000",
"is_edit_eligible": true,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"state": "Enabled"
},
"source": "<a href=\"https://www.semrush.com/\" rel=\"nofollow\">Semrush Social Media Tool</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Tue Nov 08 13:00:28 +0000 2022",
"conversation_id_str": "1589966079513673728",
"display_text_range": [
0,
201
],
"entities": {
"user_mentions": [],
"urls": [
{
"display_url": "bit.ly/3CodMVc",
"expanded_url": "https://bit.ly/3CodMVc",
"url": "https://t.co/3ooXfi4xpx",
"indices": [
178,
201
]
}
],
"hashtags": [
{
"indices": [
157,
169
],
"text": "webscraping"
},
{
"indices": [
170,
176
],
"text": "proxy"
}
],
"symbols": []
},
"favorite_count": 0,
"favorited": false,
"full_text": "Best Proxy Providers for Web Scraping \ud83e\udd16\n\nWhat do we cover in this article?\ud83d\udc47\n\n\ud83d\udd77 Quality Evaluation\n\ud83d\udd77 Pricing Evaluation\n\ud83d\udd77 Evaluation Methodology\n\ud83d\udd77 Providers\n\n#webscraping #proxy\n\nhttps://t.co/3ooXfi4xpx",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 1,
"retweet_count": 0,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1589966079513673728"
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
},
{
"__typename": "Tweet",
"rest_id": "1587431468141318146",
"core": {
"user_results": {
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
},
"card": {
"rest_id": "https://t.co/JLCeZGeDtF",
"legacy": {
"binding_values": [
{
"key": "photo_image_full_size_large",
"value": {
"image_value": {
"height": 419,
"width": 800,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=800x419"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image",
"value": {
"image_value": {
"height": 146,
"width": 280,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=280x150"
},
"type": "IMAGE"
}
},
{
"key": "description",
"value": {
"string_value": "Introduction to proxy usage in web scraping. What types of proxies are there? How to evaluate proxy providers and avoid common issues.",
"type": "STRING"
}
},
{
"key": "domain",
"value": {
"string_value": "scrapfly.io",
"type": "STRING"
}
},
{
"key": "thumbnail_image_large",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=600x600"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image_small",
"value": {
"image_value": {
"height": 202,
"width": 386,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=386x202"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_original",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=orig"
},
"type": "IMAGE"
}
},
{
"key": "site",
"value": {
"scribe_key": "publisher_id",
"type": "USER",
"user_value": {
"id_str": "1310623081300402178",
"path": []
}
}
},
{
"key": "photo_image_full_size_small",
"value": {
"image_value": {
"height": 202,
"width": 386,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=386x202"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image_large",
"value": {
"image_value": {
"height": 419,
"width": 800,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=800x419"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_small",
"value": {
"image_value": {
"height": 75,
"width": 144,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=144x144"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_x_large",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "photo_image_full_size_original",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=orig"
},
"type": "IMAGE"
}
},
{
"key": "vanity_url",
"value": {
"scribe_key": "vanity_url",
"string_value": "scrapfly.io",
"type": "STRING"
}
},
{
"key": "photo_image_full_size",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=600x314"
},
"type": "IMAGE"
}
},
{
"key": "thumbnail_image_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 171,
"red": 86
},
"percentage": 98.74
},
{
"rgb": {
"blue": 247,
"green": 211,
"red": 167
},
"percentage": 1.26
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "title",
"value": {
"string_value": "Introduction To Proxies in Web Scraping",
"type": "STRING"
}
},
{
"key": "summary_photo_image_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 171,
"red": 86
},
"percentage": 98.74
},
{
"rgb": {
"blue": 247,
"green": 211,
"red": 167
},
"percentage": 1.26
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "summary_photo_image_x_large",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "summary_photo_image",
"value": {
"image_value": {
"height": 314,
"width": 600,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=600x314"
},
"type": "IMAGE"
}
},
{
"key": "photo_image_full_size_color",
"value": {
"image_color_value": {
"palette": [
{
"rgb": {
"blue": 241,
"green": 171,
"red": 86
},
"percentage": 98.74
},
{
"rgb": {
"blue": 247,
"green": 211,
"red": 167
},
"percentage": 1.26
}
]
},
"type": "IMAGE_COLOR"
}
},
{
"key": "photo_image_full_size_x_large",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=2048x2048_2_exp"
},
"type": "IMAGE"
}
},
{
"key": "card_url",
"value": {
"scribe_key": "card_url",
"string_value": "https://t.co/JLCeZGeDtF",
"type": "STRING"
}
},
{
"key": "summary_photo_image_original",
"value": {
"image_value": {
"height": 430,
"width": 822,
"url": "https://pbs.twimg.com/card_img/1668170244408373250/koln1H3u?format=png&name=orig"
},
"type": "IMAGE"
}
}
],
"card_platform": {
"platform": {
"audience": {
"name": "production"
},
"device": {
"name": "Swift",
"version": "12"
}
}
},
"name": "summary_large_image",
"url": "https://t.co/JLCeZGeDtF",
"user_refs_results": [
{
"result": {
"__typename": "User",
"id": "VXNlcjoxMzEwNjIzMDgxMzAwNDAyMTc4",
"rest_id": "1310623081300402178",
"affiliates_highlighted_label": {},
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {
"created_at": "Mon Sep 28 16:51:22 +0000 2020",
"default_profile": true,
"default_profile_image": false,
"description": "Web Scraping API - turn any website into a database!\n\nScrapFly allows you to quickly achieve your data goals without web scraping challenges and errors.",
"entities": {
"description": {
"urls": []
},
"url": {
"urls": [
{
"display_url": "scrapfly.io",
"expanded_url": "https://scrapfly.io",
"url": "https://t.co/1Is3k6KzyM",
"indices": [
0,
23
]
}
]
}
},
"fast_followers_count": 0,
"favourites_count": 26,
"followers_count": 163,
"friends_count": 993,
"has_custom_timelines": true,
"is_translator": false,
"listed_count": 2,
"location": "Paris",
"media_count": 11,
"name": "Scrapfly",
"normal_followers_count": 163,
"pinned_tweet_ids_str": [],
"possibly_sensitive": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/1310623081300402178/1601320645",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1310658795715076098/XedZDwC7_normal.jpg",
"profile_interstitial_type": "",
"screen_name": "Scrapfly_dev",
"statuses_count": 56,
"translator_type": "none",
"url": "https://t.co/1Is3k6KzyM",
"verified": false,
"withheld_in_countries": []
}
}
}
]
}
},
"unified_card": {
"card_fetch_state": "NoCard"
},
"edit_control": {
"edit_tweet_ids": [
"1587431468141318146"
],
"editable_until_msecs": "1667309929000",
"is_edit_eligible": true,
"edits_remaining": "5"
},
"is_translatable": false,
"views": {
"state": "Enabled"
},
"source": "<a href=\"https://www.semrush.com/\" rel=\"nofollow\">Semrush Social Media Tool</a>",
"legacy": {
"bookmark_count": 0,
"bookmarked": false,
"created_at": "Tue Nov 01 13:08:49 +0000 2022",
"conversation_id_str": "1587431468141318146",
"display_text_range": [
0,
220
],
"entities": {
"user_mentions": [],
"urls": [
{
"display_url": "bit.ly/3RNDN6e",
"expanded_url": "https://bit.ly/3RNDN6e",
"url": "https://t.co/JLCeZGeDtF",
"indices": [
197,
220
]
}
],
"hashtags": [
{
"indices": [
175,
187
],
"text": "webscraping"
},
{
"indices": [
188,
194
],
"text": "proxy"
}
],
"symbols": []
},
"favorite_count": 0,
"favorited": false,
"full_text": "Introduction To Proxies in Web Scraping \ud83d\udccd\n\nWhat do we cover in this article?\ud83d\udc47\n\n\ud83d\udd77 What's a Proxy?\n\ud83d\udd77 IP Protocol Versions - Proxy Protocols\n\ud83d\udd77 Proxy Types\n\ud83d\udd77 Common Proxy Issues\n\n#webscraping #proxy \n\nhttps://t.co/JLCeZGeDtF",
"is_quote_status": false,
"lang": "en",
"possibly_sensitive": false,
"possibly_sensitive_editable": true,
"quote_count": 0,
"reply_count": 1,
"retweet_count": 2,
"retweeted": false,
"user_id_str": "1310623081300402178",
"id_str": "1587431468141318146"
},
"quick_promote_eligibility": {
"eligibility": "IneligibleUserUnauthorized"
}
}
]
}
Now that we know how to scrape individual tweets and Twitter users let's take a look at how to find them.
The obvious way is to use Twitter's search system though to scrape Twitter's search we need to login which is not advisable when web scraping as it can lead to account suspension.
Instead, we can find the latest Tweets following one of many publicly available Twitter topic timelines.
Twitter topic timelines can be found over at topics homepage:
twitter.com/i/topics/picker/home
To scrape it we can use the same background request capturing technique and capture the TopicTimeline endpoint:
from playwright.sync_api import sync_playwright
from nested_lookup import nested_lookup
def scrape_topic(url: str) -> dict:
"""
Scrape Twitter Topic timeline for latest public tweets e.g.:
https://twitter.com/i/topics/853980498816679937
The list of Twitter topics can be found here:
https://twitter.com/i/topics/picker/home
"""
_xhr_calls = []
def intercept_response(response):
"""capture all background requests and save them"""
# we can extract details from background requests
if response.request.resource_type == "xhr":
_xhr_calls.append(response)
return response
with sync_playwright() as pw:
browser = pw.chromium.launch()
context = browser.new_context(viewport={"width": 1920, "height": 1080})
page = context.new_page()
# enable background request intercepting:
page.on("response", intercept_response)
# go to url and wait for the page to load
page.goto(url)
page.wait_for_selector("[data-testid='tweet']")
# find all tweet background requests:
topic_calls = [f for f in _xhr_calls if "TopicLandingPage" in f.url]
tweets = []
for xhr in topic_calls:
data = xhr.json()
xhr_tweets = nested_lookup("tweet_results", data)
tweets.extend([tweet["result"] for tweet in xhr_tweets])
return tweets
if __name__ == "__main__":
# example: "dogs" topic:
print(scrape_topic("https://twitter.com/i/topics/853980498816679937"))
import json
from typing import Dict
from nested_lookup import nested_lookup
from scrapfly import ScrapeConfig, ScrapflyClient
SCRAPFLY = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"])
BASE_CONFIG = {
# Twitter.com requires Anti Scraping Protection bypass feature.
# for more: https://scrapfly.io/docs/scrape-api/anti-scraping-protection
"asp": True,
# Twitter.com is javascript-powered web application so it requires
# headless browsers for scraping
"render_js": True,
"country": "CA", # set prefered country here, for example Canada
}
async def scrape_topic(url: str) -> Dict:
"""
Scrape Twitter Topic timeline for latest public tweets e.g.:
https://twitter.com/i/topics/853980498816679937
The list of Twitter topics can be found here:
https://twitter.com/i/topics/picker/home
"""
result = await scrapfly.async_scrape(ScrapeConfig(
url,
wait_for_selector="[data-testid='tweet']"
**BASE_CONFIG
))
# capture background requests and extract ones that contain user data
# and their latest tweets
_xhr_calls = result.scrape_result["browser_data"]["xhr_call"]
topic_calls = [f for f in _xhr_calls if "TopicLandingPage" in f["url"]]
tweets = []
for xhr in topic_calls:
if not xhr["response"]:
continue
data = json.loads(xhr["response"]["body"])
xhr_tweets = nested_lookup("tweet_results", data)
tweets.extend([parse_tweet(tweet["result"]) for tweet in xhr_tweets])
return tweets
Above just like before we startup a headless browser, navigate to a page and capture all background requests. Then, we find the background request that contains TopicLandingPage in the request url and parse it's response for Tweet data attached to this topic.
If we start scraping Twitter at scale we start to quickly run into blocking as Twitter doesn't allow automated requests and will block scrapers IP address after extended use.
To get around this we can use proxies and web scraping APIs such as ScrapFly.
ScrapFly is essentially a HTTP proxy on steroids which provides web scraping abstraction features like:
And much more. To use ScrapFly with Python we can take advantage of Python SDK:
from scrapfly import ScrapflyClient, ScrapeConfig
scrapfly = ScrapflyClient(key="YOUR SCRAPFLY KEY")
result = scrapfly.scrape(ScrapeConfig(
"https://twitter.com/Scrapfly_dev",
# we can enable features like:
# cloud headless browser use
render_js=True,
# anti scraping protection bypass
asp=True,
# screenshot taking
screenshots={"all": "fullpage"},
# proxy country selection
country="US",
))
For more on using ScrapFly to scrape Twitter see the Full Scraper Code section.
To wrap up this Python Twitter scraper let's take a look at some frequently asked questions regarding web scraping Twitter:
Yes, all of the data on Twitter is available publically so it's perfectly legal to scrape. However, note that some tweets can contain copyrighted material like images or videos and scraping them can be illegal (though scraping the URLs are perfectly fine).
Twitter is a complex javascript-heavy website and is hostile to web scraping so it's easy to get blocked. To avoid this you can use ScrapFly which provides anti scraping technology bypass and proxy rotation. Alternatively, see our article on how to avoid web scraper blocking.
The legality of scraping Twitter while logged in is a bit of a grey area. Generally, logging in legally binds the user to the website's terms of service which in Twitter's case forbids automated scraping. This allows Twitter to suspend your account or even take legal action. It's best to avoid web scraping Twitter while logged in if possible.
If you're using browser automation tools like Playwright (used in this article) then you can block images and unnecessary resources to save bandwidth and speed up scraping.
In this tutorial, we've taken a look at how to scrape data from Twitter using Python headless browser's through Playwright or Scrapfly SDK.
To start, we've taken a look at how Twitter works and identified where the data is located. We found that Twitter is using background request to populate Tweet, profile and timeline pages.
To capture and scrape these background request we used the intercept functions and parsed the raw datasets to something more usable using jmespath and nested_lookup libraries.
Finally, to avoid blocking we've taken a look at ScrapFly web scraping API which provides a simple way to scrape Twitter at scale using proxies and anti scraping technology bypassing. Try out ScrapFly for free!
Guide how to scrape Threads - new social media network by Meta and Instagram - using Python and popular libraries like Playwright and background request capture techniques.
Goat.com is a rising storefront for luxury fashion apparel items. It's known for high quality apparel data so in this tutorial we'll take a look how to scrape it using Python.
In this fashion scrapeguide we'll be taking a look at Fashionphile - another major 2nd hand luxury fashion marketplace. We'll be using Python and hidden web data scraping to grap all of this data in just few lines of code.
