How to Scrape StockX e-commerce Data with Python
In this first entry in our fashion data web scraping series we'll be taking a look at StockX.com - a marketplace that treats apparel as stocks and how to scrape it all.
In this web scraping tutorial, we'll be taking a look at how to scrape Zillow.com - the biggest real estate marketplace in the United States.
In this guide, we'll be scraping rent and sale property information such as pricing info, addresses, photos and phone numbers displayed on Zillow.com property pages.
We'll start with a brief overview of how the website works. Then we'll take a look at how to use the search system to discover properties and, finally, how to scrape all of the property information.
We'll be using Python with a few community packages that'll make this web scraper a breeze - let's dive in!
If you're new to web scraping with Python we recommend checking out our full introduction tutorial to web scraping with Python and common best practices.
Zillow.com contains a massive real estate dataset: prices, locations, contact information, etc. This is valuable information for market analytics, the study of the housing industry, and a general competitor overview.
So, if we know how to extract data from Zillow we can have access to the biggest real estate property dataset in the US!
For more on scraping use cases see our extensive write-up Scraping Use Cases
In this tutorial, we'll be using Python with two community packages:
Optionally we'll also use loguru - a pretty logging library that'll help us to keep track of what's going on.
These packages can be easily installed via pip install command:
$ pip install httpx parsel loguru
Alternatively, feel free to swap httpx out with any other HTTP client package such as requests as we'll only need basic HTTP functions which are almost interchangeable in every library. As for, parsel, another great alternative is the beautifulsoup package.
We'll start our python Zillow scraper by looking at how we can find property listings. For this, Zillow provides powerful search functionality.
Let's take a look at how it functions and how we can use it in Zillow web scraping with Python:
Above, we can see that once we submit our search, a background request is being made to Zillow's search API. We send a search query with some map coordinates and receive hundreds of listing previews. We can see that to query Zillow we only need a few parameter inputs:
{
"searchQueryState":{
"pagination":{},
"usersSearchTerm":"New Haven, CT",
"mapBounds":
{
"west":-73.03037621240235,
"east":-72.82781578759766,
"south":41.23043771298298,
"north":41.36611033618769
},
},
"wants": {
"cat1":["mapResults"]
},
"requestId": 2
}
We can see that this API is really powerful and allows us to find listings in any map area defined by two location points comprised of 4 direction values: north, west, south and east:
This means we can find properties of any location area as long as we know its latitude and longitude. We can replicate this request in our python scraper:
from urllib.parse import urlencode
import json
import httpx
# we should use browser-like request headers to prevent being instantly blocked
BASE_HEADERS = {
"accept-language": "en-US,en;q=0.9",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"accept-language": "en-US;en;q=0.9",
"accept-encoding": "gzip, deflate, br",
}
url = "https://www.zillow.com/search/GetSearchPageState.htm?"
parameters = {
"searchQueryState": {
"pagination": {},
"usersSearchTerm": "New Haven, CT",
# map coordinates that indicate New Haven city's area
"mapBounds": {
"west": -73.03037621240235,
"east": -72.82781578759766,
"south": 41.23043771298298,
"north": 41.36611033618769,
},
},
"wants": {
# cat1 stands for agent listings
"cat1": ["mapResults"]
# and cat2 for non-agent listings
# "cat2":["mapResults"]
},
"requestId": 2,
}
response = httpx.get(url + urlencode(parameters), headers=BASE_HEADERS)
data = response.json()
results = response.json()["cat1"]["searchResults"]["mapResults"]
print(json.dumps(results, indent=2))
print(f"found {len(results)} property results")
We can see that we can replicate this search request relatively easily. So, let's take a look at how we can scrape this properly!
To scrape Zillow's search, we need geographical location details which are difficult to come up with unless you're familiar with geographical programming. However, there's an easy way to find the location's geographical details by exploring Zillow's search page itself.
If we take a look at search URL like zillow.com/homes/New-Haven,-CT_rb/ we can see geographical details hidden away in the HTML body:
We can use simple regular expression patterns to extract these details and submit our geographically based search request. Let's see how we can do it in Python scraping code:
from loguru import logger as log
from urllib.parse import quote
import httpx
async def _search(query:str, session: httpx.AsyncClient, filters: dict=None, categories=("cat1", "cat2")):
"""base search function which is used by sale and rent search functions"""
html_response = await session.get(f"https://www.zillow.com/homes/{query}_rb/")
# find query data in search landing page
query_data = json.loads(re.findall(r'"queryState":(\{.+}),\s*"filter', html_response.text)[0])
if filters:
query_data["filterState"] = filters
# scrape search API
url = "https://www.zillow.com/search/GetSearchPageState.htm?"
found = []
# cat1 - Agent Listings
# cat2 - Other Listings
for category in categories:
full_query = {
"searchQueryState": json.dumps(query_data),
"wants": json.dumps({category: ["mapResults"]}),
"requestId": randint(2, 10),
}
api_response = await session.get(url + urlencode(full_query, quote_via=quote))
data = api_response.json()
_total = data["categoryTotals"][category]["totalResultCount"]
if _total > 500:
log.warning(f"query has more results ({_total}) than 500 result limit ")
else:
log.info(f"found {_total} results for query: {query}")
map_results = data[category]["searchResults"]["mapResults"]
found.extend(map_results)
return found
async def search_sale(query: str, session: httpx.AsyncClient):
"""search properties that are for sale"""
log.info(f"scraping sale search for: {query}")
return await _search(query=query, session=session)
async def search_rent(query: str, session: httpx.AsyncClient):
"""search properites that are for rent"""
log.info(f"scraping rent search for: {query}")
filters = {
"isForSaleForeclosure": {"value": False},
"isMultiFamily": {"value": False},
"isAllHomes": {"value": True},
"isAuction": {"value": False},
"isNewConstruction": {"value": False},
"isForRent": {"value": True},
"isLotLand": {"value": False},
"isManufactured": {"value": False},
"isForSaleByOwner": {"value": False},
"isComingSoon": {"value": False},
"isForSaleByAgent": {"value": False},
}
return await _search(query=query, session=session, filters=filters, categories=["cat1"])
Above, we define our search functions for scraping rent and sale searches. The first thing we notice is that the rent and the sale pages use the same search endpoint. The only difference is that the rent search applies extra filtering to filter out sale properties.
Let's run this Zillow data scraper and see what results we receive:
import json
import asyncio
async def run():
limits = httpx.Limits(max_connections=5)
async with httpx.AsyncClient(limits=limits, timeout=httpx.Timeout(15.0), headers=BASE_HEADERS) as session:
data = await search_rent("New Haven, CT", session)
print(json.dumps(data, indent=2))
if __name__ == "__main__":
asyncio.run(run())
[
{
"buildingId": "40.609608--73.960045",
"lotId": 1004524429,
"price": "From $295,000",
"latLong": {
"latitude": 40.609608,
"longitude": -73.960045
},
"minBeds": 1,
"minBaths": 1.0,
"minArea": 1200,
"imgSrc": "https://photos.zillowstatic.com/fp/3c0259c716fc4793a65838aa40af6350-p_e.jpg",
"hasImage": true,
"plid": "1611681",
"isFeaturedListing": false,
"unitCount": 2,
"isBuilding": true,
"address": "1625 E 13th St, Brooklyn, NY",
"variableData": {},
"badgeInfo": null,
"statusType": "FOR_SALE",
"statusText": "For Rent",
"listingType": "",
"isFavorite": false,
"detailUrl": "/b/1625-e-13th-st-brooklyn-ny-5YGKWY/",
"has3DModel": false,
"hasAdditionalAttributions": false,
},
...
]
Note: zillow's search is limited to 500 properties per search, so we need to search in smaller geographical squares or use Zillow's zipcode index, which contains all US zipcodes that essential are small geographical zones!
The search returned a lot of useful preview data about each listing. It contains fields like address, geolocation, and some metadata. Though, to retrieve all the listing data, we need to scrape each property listing page, which we can find in the detailUrl field.
So, for our scraper, we can discover properties via location name (be it city, zip code etc.), scrape property previews and then pull all detailUrl fields to scrape all property data. Next, let's take a look how can we do that.
Now that we found our listing previews, we can extract the rest of the listing information by scraping each individual page.
To start, let's take a look at where the data we want is located in the property page like the one we scraped previously: zillow.com/b/1625-e-13th-st-brooklyn-ny-5YGKWY/
If we take a page source of this page (or any other listing) we can see that property data is hidden in the HTML body as a javascript variable:
This is generally referred as "javascript state cache" and is used by various javascript front ends for dynamic data rendering.
For more on hidden data scraping see our full introduction article that this type of web scraping in greater detail
In this particular example, Zillow is using Next.js framework.
Let's add property scraping and parsing to our scraper code:
import json
import asyncio
import httpx
from parsel import Selector
from typing import List
def parse_property(data: dict) -> dict:
"""parse zillow property"""
# zillow property data is massive, let's take a look just
# at the basic information to keep this tutorial brief:
parsed = {
"address": data["address"],
"description": data["description"],
"photos": [photo["url"] for photo in data["galleryPhotos"]],
"zipcode": data["zipcode"],
"phone": data["buildingPhoneNumber"],
"name": data["buildingName"],
# floor plans include price details, availability etc.
"floor_plans": data["floorPlans"],
}
return parsed
async def scrape_properties(urls: List[str], session: httpx.AsyncClient):
"""scrape zillow properties"""
async def scrape(url):
resp = await session.get(url)
sel = Selector(text=resp.text)
data = sel.css("script#__NEXT_DATA__::text").get()
if data:
# some properties are located in NEXT DATA cache
data = json.loads(data)
return parse_property(data["props"]["initialReduxState"]["gdp"]["building"])
else:
# other times it's in Apollo cache
data = sel.css('script#hdpApolloPreloadedData::text').get()
data = json.loads(json.loads(data)['apiCache'])
property_data = next(v['property'] for k, v in data.items() if 'ForSale' in k)
return property_data
return await asyncio.gather(*[scrape(url) for url in urls])
Above, to pull data from Zillow we wrote a small function that takes a list of property URLs. Then we scrape their HTML pages, extract embedded javascript state data and parse property info such as address, prices and phone numbers!
Let's run this property scraper and see the results it generates:
async def run():
limits = httpx.Limits(max_connections=5)
async with httpx.AsyncClient(limits=limits, timeout=httpx.Timeout(15.0), headers=BASE_HEADERS) as session:
data = await scrape_properties(
["https://www.zillow.com/b/1625-e-13th-st-brooklyn-ny-5YGKWY/"],
session=session
)
print(json.dumps(data, indent=2))
if __name__ == "__main__":
asyncio.run(run())
[
{
"address": {
"streetAddress": "1065 2nd Ave",
"city": "New York",
"state": "NY",
"zipcode": "10022",
"__typename": "Address",
"neighborhood": null
},
"description": "Inspired by Alvar Aaltos iconic vase, Aalto57s sculptural architecture reflects classic concepts of design both inside and out. Each residence in this boutique rental building features clean modern finishes. Amenities such as a landscaped terrace with gas grills, private and group dining areas, sun loungers, and fire feature as well as an indoor rock climbing wall, basketball court, game room, childrens playroom, guest suite, and a fitness center make Aalto57 a home like no other.",
"photos": [
"https://photos.zillowstatic.com/fp/0c1099a1882a904acc8cedcd83ebd9dc-p_d.jpg",
"..."
],
"zipcode": "10022",
"phone": "646-681-3805",
"name": "Aalto57",
"floor_plans": [
{
"zpid": "2096631846",
"__typename": "FloorPlan",
"availableFrom": "1657004400000",
"baths": 1,
"beds": 1,
"floorPlanUnitPhotos": [],
"floorplanVRModel": null,
"maxPrice": 6200,
"minPrice": 6200,
"name": "1 Bed/1 Bath-1D",
...
}
...
]
}]
We wrote a quick python scraper that finds Zillow's properties from a given query string and then scrapes each property page for property information.
However, to run this scraper at scale without being blocked, we'll take a look at using ScrapFly web scraping API. ScrapFly will help us to scale up our scraper and avoid blocking and captcha requests.
Scraping Zillow.com data doesn't seem to be too difficult though unfortunately when scraping at scale it's very likely we'll be blocked or need solve catpchas, which will hinder or completely disable our web scraper.
To get around this, let's take advantage of ScrapFly API which can avoid all of these blocks for us!
ScrapFly offers several powerful features that'll help us to get around Zillow's web scraper blocking:
For this we'll be using scrapfly-sdk python package. First, let's install scrapfly-sdk using pip:
$ pip install scrapfly-sdk
To take advantage of ScrapFly's API in our Zillow web scraper all we need to do is change our httpx session code with scrapfly-sdk client requests:
import httpx
response = httpx.get("some zillow url")
# in ScrapFly SDK becomes
from scrapfly import ScrapflyClient, ScrapeConfig
client = ScrapflyClient("YOUR SCRAPFLY KEY")
result = client.scrape(ScrapeConfig(
"some zillow url",
# we can select specific proxy country
country="US",
# and enable anti scraping protection bypass:
asp=True
))
For more on how to scrape data from Zillow using ScrapFly, see the Full Scraper Code section.
To wrap this guide up, let's take a look at some frequently asked questions about web scraping Zillow data:
Yes. Zillow's data is publicly available; we're not extracting anything personal or private. Scraping Zillow.com at slow, respectful rates would fall under the ethical scraping definition.
That being said, attention should be paid to GDRP compliance in the EU when scraping personal data of non-agent listings (seller's name, phone number etc). For more, see our Is Web Scraping Legal? article.
Yes, but it's extremely limited and not suitable for dataset collection and there are no Zillow API Python clients available. Instead, we can scrape Zillow data with Python and httpx, which is perfectly legal and easy to do.
We can easily create a Zillow crawler with the subjects we've covered in this tutorial. Instead of searching for properties explicitly, we can crawl Zillow properties from seed links (any Zillow URLs) and follow the related properties mentioned in a loop. For more on crawling, see How to Crawl the Web with Python.
In this tutorial we dove into Zillow data extraction by building a scraper in Python.
We used search to discover real estate properties for sale or rent in any given region. To scrape the property data, such as price and building information, contact details etc. we used hidden web data scraping by extracting Zillow's state cache from the HTML page.
For this, we used Python with httpx and parsel packages and to avoid being blocked we used ScrapFly's API that smartly configures every web scraper connection to avoid being blocked. For more on ScrapFly, see our documentation and try it out for free!
Let's take a look at how our full scraper code would look with ScrapFly integration:
import asyncio
import json
import re
from random import randint
from typing import List
from urllib.parse import urlencode, quote
from loguru import logger as log
from parsel import Selector
from scrapfly import ScrapeConfig, ScrapflyClient
async def _search(query: str, session: ScrapflyClient, filters: dict = None, categories=("cat1", "cat2")) -> List[dict]:
"""base search function which is used by sale and rent search functions"""
html_result = await session.async_scrape(
ScrapeConfig(
url=f"https://www.zillow.com/homes/{query}_rb/",
proxy_pool="public_residential_pool",
country="US",
asp=True,
)
)
query_data = json.loads(re.findall(r'"queryState":(\{.+}),\s*"filter', html_result.content)[0])
if filters:
query_data["filterState"] = filters
url = "https://www.zillow.com/search/GetSearchPageState.htm?"
found = []
# cat1 - Agent Listings
# cat2 - Other Listings
for category in categories:
full_query = {
"searchQueryState": json.dumps(query_data),
"wants": json.dumps({category: ["mapResults"]}),
"requestId": randint(2, 10),
}
api_result = await session.async_scrape(
ScrapeConfig(
url=url + urlencode(full_query, quote_via=quote),
proxy_pool="public_residential_pool",
country="US",
asp=True,
)
)
data = json.loads(api_result.content)
_total = data["categoryTotals"][category]["totalResultCount"]
if _total > 500:
log.warning(f"query has more results ({_total}) than 500 result limit ")
else:
log.info(f"found {_total} results for query: {query}")
map_results = data[category]["searchResults"]["mapResults"]
found.extend(map_results)
return found
async def search_sale(query: str, session: ScrapflyClient) -> List[dict]:
"""search properties that are for sale"""
log.info(f"scraping sale search for: {query}")
return await _search(query=query, session=session)
async def search_rent(query: str, session: ScrapflyClient) -> List[dict]:
"""search properites that are for rent"""
log.info(f"scraping rent search for: {query}")
filters = {
"isForSaleForeclosure": {"value": False},
"isMultiFamily": {"value": False},
"isAllHomes": {"value": True},
"isAuction": {"value": False},
"isNewConstruction": {"value": False},
"isForRent": {"value": True},
"isLotLand": {"value": False},
"isManufactured": {"value": False},
"isForSaleByOwner": {"value": False},
"isComingSoon": {"value": False},
"isForSaleByAgent": {"value": False},
}
return await _search(query=query, session=session, filters=filters, categories=["cat1"])
def parse_property(data: dict) -> dict:
"""parse zillow property"""
# zillow property data is massive, let's take a look just
# at the basic information to keep this tutorial brief:
parsed = {
"address": data["address"],
"description": data["description"],
"photos": [photo["url"] for photo in data["galleryPhotos"]],
"zipcode": data["zipcode"],
"phone": data["buildingPhoneNumber"],
"name": data["buildingName"],
# floor plans include price details, availability etc.
"floor_plans": data["floorPlans"],
}
return parsed
async def scrape_properties(urls: List[str], session: ScrapflyClient):
"""scrape zillow properties"""
async def scrape(url):
result = await session.async_scrape(
ScrapeConfig(url=url, asp=True, country="US", proxy_pool="public_residential_pool")
)
response = result.upstream_result_into_response()
sel = Selector(text=response.text)
data = sel.css("script#__NEXT_DATA__::text").get()
if data:
# some properties are located in NEXT DATA cache
data = json.loads(data)
return parse_property(data["props"]["initialReduxState"]["gdp"]["building"])
else:
# other times it's in Apollo cache
data = sel.css('script#hdpApolloPreloadedData::text').get()
data = json.loads(json.loads(data)['apiCache'])
property_data = next(v['property'] for k, v in data.items() if 'ForSale' in k)
return property_data
return await asyncio.gather(*[scrape(url) for url in urls])
async def run():
with ScrapflyClient(key="YOUR_SCRAPFLY_KEY", max_concurrency=2) as session:
rentals = await search_rent("New Haven, CT", session)
sales = await search_sale("New Haven, CT", session)
property_data = await scrape_properties(
["https://www.zillow.com/b/aalto57-new-york-ny-5twVDd/"], session=session
)
if __name__ == "__main__":
asyncio.run(run())
In this first entry in our fashion data web scraping series we'll be taking a look at StockX.com - a marketplace that treats apparel as stocks and how to scrape it all.
With the news of Twitter dropping free API access we're taking a look at web scraping Twitter using Python for free. In this tutorial we'll cover two methods: using Playwright and Twitter's hidden graphql API.
In this scrape guide we'll be taking a look at scraping RightMove.co.uk - one of the most popular real estate listing websites in the United Kingdom. We'll be scraping hidden web data and backend APIs directly using Python.
