How to Scrape Zillow Real Estate Property Data in Python

In this web scraping tutorial, we'll be taking a look at how to scrape Zillow.com - the biggest real estate marketplace in the United States.

In this guide, we'll be scraping rent and sale property information such as pricing info, addresses, photos and phone numbers displayed on Zillow.com property pages.
We'll start with a brief overview of how the website works. Then we'll take a look at how to use the search system to discover properties and, finally, how to scrape all of the property information.

We'll be using Python with a few community packages that'll make this web scraper a breeze - let's dive in!

Why Scrape Zillow.com?

Zillow.com contains a massive real estate dataset: prices, locations, contact information, etc. This is valuable information for market analytics, the study of the housing industry, and a general competitor overview.

So, if we know how to extract data from Zillow we can have access to the biggest real estate property dataset in the US!

For more on scraping use cases see our extensive write-up Scraping Use Cases

Project Setup

In this tutorial, we'll be using Python with two community packages:

• httpx - HTTP client library which will let us communicate with Zillow.com's servers
• parsel - HTML parsing library which will help us to parse our web scraped HTML files.

Optionally we'll also use loguru - a pretty logging library that'll help us to keep track of what's going on.

These packages can be easily installed via pip install command:

$ pip install httpx parsel loguru

Alternatively, feel free to swap httpx out with any other HTTP client package such as requests as we'll only need basic HTTP functions which are almost interchangeable in every library. As for, parsel, another great alternative is the beautifulsoup package.

Hands on Python Web Scraping Tutorial and Example Project

If you're new to web scraping with Python we recommend checking out our full introduction tutorial to web scraping with Python and common best practices.

Scraping Zillow Properties

To start, let's take a look at thow to scrape Zillow property dataset from given Zillow page urls.

First, let's take a look at where the data we want is located in the property page, like this one: zillow.com/b/1625-e-13th-st-brooklyn-ny-5YGKWY/

If we take a look at the property listing's page source, we can see that the property dataset is hidden in the HTML body as a javascript variable:

This is generally referred to as hidden web data scraping. In this example, Zillow's backend stores property dataset to a javascript variable so the front-end designers can access it to display it for the end users.

In this particular example, Zillow is using Next.js framework.

Let's add property scraping and parsing to our scraper code:

import asyncio
from typing import List
import httpx
import json

from parsel import Selector

client = httpx.AsyncClient(
    # enable http2
    http2=True,
    # add basic browser like headers to prevent being blocked
    headers={
        "accept-language": "en-US,en;q=0.9",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
        "accept-language": "en-US;en;q=0.9",
        "accept-encoding": "gzip, deflate, br",
    },
)


async def scrape_properties(urls: List[str]):
    """scrape zillow property pages for property data"""
    to_scrape = [client.get(url) for url in urls]
    results = []
    for response in asyncio.as_completed(to_scrape):
        response = await response
        assert response.status_code == 200, "request has been blocked"
        selector = Selector(response.text)
        data = selector.css("script#__NEXT_DATA__::text").get()
        if data:
            # Option 1: some properties are located in NEXT DATA cache
            data = json.loads(data)
            property_data = json.loads(data["props"]["pageProps"]["gdpClientCache"])
            property_data = next(v['property'] for v in property_data.values())
        else:
            # Option 2: other times it's in Apollo cache
            data = selector.css("script#hdpApolloPreloadedData::text").get()
            data = json.loads(json.loads(data)["apiCache"])
            property_data = next(
                v["property"] for k, v in data.items() if "ForSale" in k
            )
        results.append(property_data)
    return results

# example run:
if __name__ == "__main__":
    async def run():
        data = await scrape_properties(
            ["https://www.zillow.com/homedetails/1625-E-13th-St-APT-3K-Brooklyn-NY-11229/245001606_zpid/"],
        )
        print(json.dumps(data, indent=2))
    asyncio.run(run())

Above, to pull data from Zillow we wrote a small function that takes a list of property URLs. Then we scrape their HTML pages, extract embedded javascript state data and parse property info such as address, prices and phone numbers!

Let's run this property scraper and see the results it generates:

[
  {
    "address": {
      "streetAddress": "1065 2nd Ave",
      "city": "New York",
      "state": "NY",
      "zipcode": "10022",
      "__typename": "Address",
      "neighborhood": null
    },
    "description": "Inspired by Alvar Aaltos iconic vase, Aalto57s sculptural architecture reflects classic concepts of design both inside and out. Each residence in this boutique rental building features clean modern finishes. Amenities such as a landscaped terrace with gas grills, private and group dining areas, sun loungers, and fire feature as well as an indoor rock climbing wall, basketball court, game room, childrens playroom, guest suite, and a fitness center make Aalto57 a home like no other.",
    "photos": [
      "https://photos.zillowstatic.com/fp/0c1099a1882a904acc8cedcd83ebd9dc-p_d.jpg",
      "..."
    ],
    "zipcode": "10022",
    "phone": "646-681-3805",
    "name": "Aalto57",
    "floor_plans": [
      {
        "zpid": "2096631846",
        "__typename": "FloorPlan",
        "availableFrom": "1657004400000",
        "baths": 1,
        "beds": 1,
        "floorPlanUnitPhotos": [],
        "floorplanVRModel": null,
        "maxPrice": 6200,
        "minPrice": 6200,
        "name": "1 Bed/1 Bath-1D",
        ...
      }
    ...
  ]
}]

Finding Zillow Properties

Now that we know how to scrape a single Zillow property listing we can take a look at how to find the listings using Zillow's Search bar.

Let's take a look at how it functions and how we can use it in Zillow web scraping with Python:

Above, we can see that once we submit our search, a background request is being made to Zillow's search API. We send a search query with some map coordinates and receive hundreds of listing previews. We can see that to query Zillow we only need a few parameter inputs:

{
  "searchQueryState":{
    "pagination":{},
    "usersSearchTerm":"New Haven, CT",
    "mapBounds":
      {
        "west":-73.03037621240235,
        "east":-72.82781578759766,
        "south":41.23043771298298,
        "north":41.36611033618769
      },
    },
  "wants": {
    "cat1":["mapResults"]
  },
  "requestId": 2
}

We can see that this API is really powerful and allows us to find listings in any map area defined by two location points comprised of 4 direction values: north, west, south and east:

This means we can find properties of any location area as long as we know its latitude and longitude. We can replicate this request in our python scraper:

from urllib.parse import urlencode
import json
import httpx

# we should use browser-like request headers to prevent being instantly blocked
BASE_HEADERS = {
    "accept-language": "en-US,en;q=0.9",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
    "accept-language": "en-US;en;q=0.9",
    "accept-encoding": "gzip, deflate, br",
}


url = "https://www.zillow.com/search/GetSearchPageState.htm?"
parameters = {
    "searchQueryState": {
        "pagination": {},
        "usersSearchTerm": "New Haven, CT",
        # map coordinates that indicate New Haven city's area
        "mapBounds": {
            "west": -73.03037621240235,
            "east": -72.82781578759766,
            "south": 41.23043771298298,
            "north": 41.36611033618769,
        },
    },
    "wants": {
        "cat1": ["listResults", "mapResults"], "cat2": ["total"]
    },
    "requestId": 2,
}
response = httpx.get(url + urlencode(parameters), headers=BASE_HEADERS)
assert response.status_code == 200, "request has been blocked"
data = response.json()
results = response.json()["cat1"]["searchResults"]["mapResults"]
print(json.dumps(results, indent=2))
print(f"found {len(results)} property results")

We can see that we can replicate this search request relatively easily. So, let's take a look at how we can scrape this properly!

Scraping Zillow Search

To scrape Zillow's search, we need geographical location details which are difficult to come up with unless you're familiar with geographical programming. However, there's an easy way to find the location's geographical details by exploring Zillow's search page itself.
If we take a look at search URL like zillow.com/homes/New-Haven,-CT_rb/ we can see geographical details hidden away in the HTML body:

We can use simple regular expression patterns to extract these details and submit our geographically based search request. Let's see how we can do it in Python scraping code:

from loguru import logger as log
from urllib.parse import quote
import httpx

async def _search(query:str, session: httpx.AsyncClient, filters: dict=None, categories=("cat1", "cat2")):
    """base search function which is used by sale and rent search functions"""
    html_response = await session.get(f"https://www.zillow.com/homes/{query}_rb/")
    # find query data in search landing page
    query_data = json.loads(re.findall(r'"queryState":(\{.+}),\s*"filter', html_response.text)[0])
    if filters:
        query_data["filterState"] = filters

    # scrape search API
    url = "https://www.zillow.com/search/GetSearchPageState.htm?"
    found = []
    # cat1 - Agent Listings
    # cat2 - Other Listings
    for category in categories:
        full_query = {
            "searchQueryState": json.dumps(query_data),
            "wants": json.dumps({category: ["mapResults"]}),
            "requestId": randint(2, 10),
        }
        api_response = await session.get(url + urlencode(full_query, quote_via=quote))
        data = api_response.json()
        _total = data["categoryTotals"][category]["totalResultCount"]
        if _total > 500:
            log.warning(f"query has more results ({_total}) than 500 result limit ")
        else:
            log.info(f"found {_total} results for query: {query}")
        map_results = data[category]["searchResults"]["mapResults"]
        found.extend(map_results)
    return found


async def search_sale(query: str, session: httpx.AsyncClient):
    """search properties that are for sale"""
    log.info(f"scraping sale search for: {query}")
    return await _search(query=query, session=session)


async def search_rent(query: str, session: httpx.AsyncClient):
    """search properites that are for rent"""
    log.info(f"scraping rent search for: {query}")
    filters = {
        "isForSaleForeclosure": {"value": False},
        "isMultiFamily": {"value": False},
        "isAllHomes": {"value": True},
        "isAuction": {"value": False},
        "isNewConstruction": {"value": False},
        "isForRent": {"value": True},
        "isLotLand": {"value": False},
        "isManufactured": {"value": False},
        "isForSaleByOwner": {"value": False},
        "isComingSoon": {"value": False},
        "isForSaleByAgent": {"value": False},
    }
    return await _search(query=query, session=session, filters=filters, categories=["cat1"])

Above, we define our search functions for scraping rent and sale searches. The first thing we notice is that the rent and the sale pages use the same search endpoint. The only difference is that the rent search applies extra filtering to filter out sale properties.

Let's run this Zillow data scraper and see what results we receive:

import json
import asyncio

async def run():
    limits = httpx.Limits(max_connections=5)
    async with httpx.AsyncClient(limits=limits, timeout=httpx.Timeout(15.0), headers=BASE_HEADERS) as session:
        data = await search_rent("New Haven, CT", session)
        print(json.dumps(data, indent=2))


if __name__ == "__main__":
    asyncio.run(run())

[
  {
    "buildingId": "40.609608--73.960045",
    "lotId": 1004524429,
    "price": "From $295,000",
    "latLong": {
      "latitude": 40.609608,
      "longitude": -73.960045
    },
    "minBeds": 1,
    "minBaths": 1.0,
    "minArea": 1200,
    "imgSrc": "https://photos.zillowstatic.com/fp/3c0259c716fc4793a65838aa40af6350-p_e.jpg",
    "hasImage": true,
    "plid": "1611681",
    "isFeaturedListing": false,
    "unitCount": 2,
    "isBuilding": true,
    "address": "1625 E 13th St, Brooklyn, NY",
    "variableData": {},
    "badgeInfo": null,
    "statusType": "FOR_SALE",
    "statusText": "For Rent",
    "listingType": "",
    "isFavorite": false,
    "detailUrl": "/b/1625-e-13th-st-brooklyn-ny-5YGKWY/",
    "has3DModel": false,
    "hasAdditionalAttributions": false,
  },
...
]

The search returned a lot of useful preview data about each listing. It contains fields like address, geolocation, and some metadata. Though, to retrieve all the listing data, we need to scrape each property listing page, which we can find in the detailUrl field.
So, for our scraper, we can discover properties via location name (be it city, zip code etc.), scrape property previews and then pull all detailUrl fields to scrape all property data. Next, let's take a look how can we do that.

We wrote a quick python scraper that finds Zillow's properties from a given query string and then scrapes each property page for property information.

However, to run this scraper at scale without being blocked, we'll take a look at using ScrapFly web scraping API. ScrapFly will help us to scale up our scraper and avoid blocking and captcha requests.

Bypass Zillow Blocking with Scrapfly

Scraping Zillow.com data doesn't seem to be too difficult though unfortunately when scraping at scale it's very likely we'll be blocked or need solve catpchas, which will hinder or completely disable our web scraper.

To get around this, let's take advantage of ScrapFly API which can avoid all of these blocks for us!

ScrapFly offers several powerful features that'll help us to get around Zillow's web scraper blocking:

• Anti Scraping Protection Bypass
• Javascript Rendering
• 190M Pool of Residential or Mobile Proxies

For this we'll be using scrapfly-sdk python package. First, let's install scrapfly-sdk using pip:

$ pip install scrapfly-sdk

To take advantage of ScrapFly's API in our Zillow web scraper all we need to do is change our httpx session code with scrapfly-sdk client requests:

import httpx

response = httpx.get("some zillow url")
# in ScrapFly SDK becomes
from scrapfly import ScrapflyClient, ScrapeConfig
client = ScrapflyClient("YOUR SCRAPFLY KEY")
result = client.scrape(ScrapeConfig(
    "some zillow url",
    # we can select specific proxy country
    country="US",
    # and enable anti scraping protection bypass:
    asp=True
))

For more on how to scrape data from Zillow using ScrapFly, see our Github collection:

FAQ

To wrap this guide up, let's take a look at some frequently asked questions about web scraping Zillow data:

Is it legal to scrape Zillow.com?

Yes. Zillow's data is publicly available; we're not extracting anything personal or private. Scraping Zillow.com at slow, respectful rates would fall under the ethical scraping definition.
That being said, attention should be paid to GDRP compliance in the EU when scraping personal data of non-agent listings (seller's name, phone number etc). For more, see our Is Web Scraping Legal? article.

Does Zillow.com have an API?

Yes, but it's extremely limited and not suitable for dataset collection and there are no Zillow API Python clients available. Instead, we can scrape Zillow data with Python and httpx, which is perfectly legal and easy to do.

How to crawl Zillow?

We can easily create a Zillow crawler with the subjects we've covered in this tutorial. Instead of searching for properties explicitly, we can crawl Zillow properties from seed links (any Zillow URLs) and follow the related properties mentioned in a loop. For more on crawling, see How to Crawl the Web with Python.

Zillow Scraping Summary

In this tutorial, we dove into Zillow data extraction by building a scraper in Python.

We used search to discover real estate properties for sale or rent in any given region. To scrape the property data, such as price and building information, contact details etc. we used hidden web data scraping by extracting Zillow's state cache from the HTML page.

For this, we used Python with httpx and parsel packages and to avoid being blocked we used ScrapFly's API that smartly configures every web scraper connection to avoid being blocked. For more on ScrapFly, see our documentation and try it out for free!