Immoscout24.ch is a popular website for real estate ads in Switzerland, which includes real estate properties for buying or renting.
In this article, we'll explore how to scrape immoscout2.ch to get property listing data. We'll also explain how to avoid Immoscout24.ch web scraping blocking. Let's dive in!
This tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect and here's a good summary of what not to do:
Scrapfly does not offer legal advice but these are good general rules to follow in web scraping
and for more you should consult a lawyer.
Scraping Immoscout24.ch opens the door to comprehensive property listing data. Allowing sellers and buyers to track market trends including property prices, demand and supply changes over time.
Real estate data from scraping immopscout24.ch can help investors identify underpriced properties or areas with potential growth, leading to better decision-making.
Moreover, manually exploring real estate data from websites can be tedious and time-consuming. Therefore, web scraping immoscout24.ch can save a lot of manual effort by quickly retrieving thousands of listings.
To scrape immoscout24.ch, we'll use a few Python libraries.
httpx - for sending HTTP requests to the website.parsel - for parsing HTML using XPath and CSS Selectors.scrapfly-sdk - for avoiding immoscout24.ch scraping blocking using the ScrapFly web scraping API.asyncio - for increasing web scraping speed by running our code asynchronously.Python already includes asyncio and the rest can be installed using the following pip command:
pip install httpx parsel scrapfly-sdk
Let's start by scraping immoscout24.ch search pages which contain property preview data and links to full property listing details. For our example let's take this search url immoscout24.ch/en/real-estate/rent/city-bern:
We can manipulate the search results by changing the parts of URL like URL parameters or location:
pn parameter at the end of the URL with the desired page number.city-bern with the desired city name.rent or buy in the search URL.Instead of scraping immoscout24.ch property listings by parsing the visible HTML elements, we'll extract hidden JSON proprety datasets directly from the invisible script tags. This type of data is typically known as hidden web data, which is the source of HTML data we see on the page.
To view this data in the HTML, open the browser developer tools by clicking the F12 key. Then, scroll down to the script tag that has an HTML similar to this:
To scrape this data, we'll select and parse this script tag data:
import asyncio
import json
from typing import List, Dict
from httpx import AsyncClient, Response
from parsel import Selector
client = AsyncClient(
headers={
# use same headers as a popular web browser (Chrome on Windows in this case)
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Language": "en-US,en;q=0.9",
}
)
def parse_next_data(response: Response) -> Dict:
"""parse listing data from script tags"""
print(response)
selector = Selector(response.text)
# extract data in JSON from the script tag
next_data = selector.xpath("//script[contains(text(),'INITIAL_STATE')]/text()").get().strip("window.__INITIAL_STATE__=")
# replace undefined values as Python doesn't recognize it
next_data = next_data.replace("undefined", "null")
if not next_data:
return
next_data_json = json.loads(next_data)
return next_data_json
async def scrape_search(url: str, max_pages: int) -> List[Dict]:
"""scrape listing data from immoscout24 search pages"""
# scrape the first search page first
first_page = await client.get(url)
data = parse_next_data(first_page)["resultList"]["search"]["fullSearch"]["result"]
search_data = data["listings"]
# get the number of maximum search pages available
max_search_pages = data["resultCount"]
print(f"scraped first search page, remaining ({max_search_pages} search pages)")
# get the number of max pages to scrape
if max_pages and max_pages < max_search_pages:
max_search_pages = max_pages
# add the remaining search pages to a scraping list
other_pages = [
client.get(url=str(first_page.url) + f"?pn={page}")
for page in range(2, max_search_pages + 1)
]
# scrape the remaining search pages concurrently
for response in asyncio.as_completed(other_pages):
data = parse_next_data(await response)
search_data.extend(data["resultList"]["search"]["fullSearch"]["result"]["listings"])
print(f"scraped {len(search_data)} property listings from search")
return search_data
import asyncio
import json
from typing import List, Dict
from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse
scrapfly = ScrapflyClient(key="Your ScrapFly API key")
def parse_next_data(response: ScrapeApiResponse) -> Dict:
"""parse listing data from script tags"""
selector = response.selector
# extract data in JSON from the script tag
next_data = next_data = selector.xpath("//script[contains(text(),'INITIAL_STATE')]/text()").get().strip("window.__INITIAL_STATE__=")
# replace undefined values as Python doesn't recognize it
next_data = next_data.replace("undefined", "null")
if not next_data:
return
next_data_json = json.loads(next_data)
return next_data_json
async def scrape_search(url: str) -> List[Dict]:
"""scrape listing data from immoscout24 search pages"""
# scrape the first search page first
first_page = await scrapfly.async_scrape(ScrapeConfig(url, asp=True, country="CH"))
data = parse_next_data(first_page)["resultList"]["search"]["fullSearch"]["result"]
search_data = data["listings"]
# get the number of maximum search pages available
max_search_pages = data["resultCount"]
print(f"scraped first search page, remaining ({max_search_pages} search pages)")
# add the remaining search pages in a scraping list
other_pages = [
ScrapeConfig(first_page.context['url']+ f"?pn={page}", asp=True, country="CH")
for page in range(2, max_search_pages + 1)
]
# scrape the remaining search pages concurrently
async for response in scrapfly.concurrent_scrape(other_pages):
data = parse_next_data(response)
search_data.extend(data["resultList"]["search"]["fullSearch"]["result"]["listings"])
print(f"scraped {len(search_data)} property listings from search")
return search_data
if __name__ == "__main__":
search_data = asyncio.run(
scrape_search(
url="https://www.immoscout24.ch/en/real-estate/rent/city-bern",
max_pages=3
)
)
# print the result in JSON format
print(json.dumps(search_data, indent=2))
Here, we create an async httpx client with a few HTTP headers to mimic a normal web browser. Then, we create a parse_next_data function to find the script tag that contains the property JSON data and clean it up. Next, we create a scrape_search function to scrape the first search page data and the maximum number of search pages available. Finally, we scrape the remaining of the pages concurrently.
The result is a list containing real estate property listings found on all search pages. Similar to this example:
[
{
"id": 7778901,
"accountId": 65708,
"companyId": 27717,
"memberPackageId": 4,
"agency": {
"companyCity": "Liebefeld",
"companyName1": "Adlatus AG",
"companyPhoneBusiness": "031 537 22 88",
"companyStreet": "Hohle Gasse 4",
"companyZip": "3097",
"firstName": "Yannick",
"gender": "m",
"showLogoOnSerp": true,
"lastName": "Morf",
"logoUrl": "https://cis01.immoscout24.ch/memberlogos/L5962-R.png",
"logoUrlDetailPage": "https://cis01.immoscout24.ch/memberlogos/L5962-R.png",
"nameFormatted": "Mr Yannick Morf",
"isAccountMigrated": true,
"isGuest": false,
"userType": "M"
},
"availableFrom": "2023-11-01T00:00:00+01:00",
"availableFromFormatted": "Immediately",
"cityId": 295,
"cityName": "Bern",
"commuteTimes": {
"defaultPois": [
{
"defaultPoiId": 2,
"label": "Bern station",
"transportations": [
{
"transportationTypeId": 6,
"travelTime": 566,
"isReachable": true
},
{
"transportationTypeId": 10,
"travelTime": 591,
"isReachable": true
},
{
"transportationTypeId": 3,
"travelTime": 575,
"isReachable": true
}
]
}
]
},
"countryId": 1,
"extraPrice": 190,
"extraPriceFormatted": "CHF 190.\u2014",
"geoAccuracy": 8,
"grossPrice": 1990,
"grossPriceFormatted": "CHF 1990.\u2014",
"hasNewBuildingProject": false,
"hasVirtualTour": false,
"isHighlighted": false,
"isNeubauLite": true,
"isNeubauLitePremium": true,
"isHgCrosslisting": false,
"isNew": false,
"isNewEndDate": "2023-10-20T11:07:50+02:00",
"isOnline": true,
"isTopListing": false,
"isPremiumToplisting": true,
"lastModified": "2023-11-06T06:47:17+01:00",
"latitude": 46.961742139557,
"longitude": 7.450882241067,
"netPrice": 1800,
"netPriceFormatted": "CHF 1800.\u2014",
"normalizedPrice": 1990,
"normalizedPriceFormatted": "CHF 1990.\u2014",
"numberOfRooms": 2.5,
"numberOfRoomsFormatted": "2.5 rooms",
"offerTypeId": 1,
"priceUnitId": 6,
"priceUnitLabel": "month",
"price": 1990,
"priceFormatted": "CHF 1990.\u2014",
"propertyDetailUrl": "/en/d/duplex-maisonette-rent-bern/7778901?s=1&t=1&l=436&ct=494&ci=1&pn=1",
"propertyUrl": "/en/d/duplex-maisonette-rent-bern/7778901",
"searchParameters": {
"s": "1",
"t": "1",
"l": "436",
"inp": "1",
"ct": "494",
"ci": "1",
"pn": "1"
},
"propertyCategoryId": 1,
"propertyTypeId": 24,
"stateShort": "BE",
"state": "Bern",
"street": "Standstrasse 21",
"surfaceLiving": 68,
"surfaceLivingFormatted": "68 m\u00b2",
"title": "urB\u00c4RN - modern living",
"videoViewingEnabled": false,
"zip": "3014",
"zipId": 864,
"packageType": 0,
"shortDescription": "Im beliebten \"Breitsch-Quartier\" entstehen 5 exklusive Duplex-Wohnungen.Das Projekt unter dem Namen urB\u00c4RN unterstreicht den Stil und das Konzept dieses Neubaus -\u00a0urbanes\u00a0Leben in\u00a0B\u00e4rn.Das Lebensgef\u00fchl im Breitenrainquartier ist einzigartig und bei den Stadtbernern \u00e4usserst beliebt. Um dieses Gef\u00fchl\u00a0zu unterstreichen, haben wir die passenden Wohnungen erstellt.modern-geradlinig-exklusiv |\u00a0urbane Architektur mit h\u00f6chster Qualit\u00e4tBei den Bildern im Inserat handelt es sich um Visualisierungen.Wir freuen uns auf Ihre Kontaktaufnahme.Adlatus AG",
"listingLayout": 3,
"packageScore": 0,
"isQualityListing": true,
"images": [
{
"url": "https://cis01.immoscout24.ch/urbaern---modern-living-bern?{width}x{height}/{resizemode}/{quality}/is24media/2a/a1/9da96f474f-230525.jpg",
"originalWidth": 3000,
"originalHeight": 2496,
"id": 106849019,
"title": "",
"description": "",
"sortOrder": 1,
"lastModified": "2023-05-25T10:38:35+02:00"
},
{
"url": "https://cis01.immoscout24.ch/urbaern---modern-living-bern?{width}x{height}/{resizemode}/{quality}/is24media/cf/25/da205ca840-230525.jpg",
"originalWidth": 3000,
"originalHeight": 2496,
"id": 106849021,
"title": "",
"description": "",
"sortOrder": 2,
"lastModified": "2023-05-25T10:38:51+02:00"
},
{
"url": "https://cis01.immoscout24.ch/urbaern---modern-living-bern?{width}x{height}/{resizemode}/{quality}/is24media/04/be/9cb85e4f4c-230525.jpg",
"originalWidth": 3000,
"originalHeight": 1875,
"id": 106849022,
"title": "",
"description": "",
"sortOrder": 3,
"lastModified": "2023-05-25T10:38:55+02:00"
},
{
"url": "https://cis01.immoscout24.ch/urbaern---modern-living-bern?{width}x{height}/{resizemode}/{quality}/is24media/64/d8/6c73d96746-230525.jpg",
"originalWidth": 3000,
"originalHeight": 1875,
"id": 106849023,
"title": "",
"description": "",
"sortOrder": 4,
"lastModified": "2023-05-25T10:39:07+02:00"
},
{
"url": "https://cis01.immoscout24.ch/urbaern---modern-living-bern?{width}x{height}/{resizemode}/{quality}/is24media/a2/ef/75a3c0d840-230525.jpg",
"originalWidth": 2494,
"originalHeight": 2880,
"id": 106849024,
"title": "",
"description": "",
"sortOrder": 5,
"lastModified": "2023-05-25T10:39:27+02:00"
},
{
"url": "https://cis01.immoscout24.ch/urbaern---modern-living-bern?{width}x{height}/{resizemode}/{quality}/is24media/61/5a/9cbe3d0043-230525.jpg",
"originalWidth": 3000,
"originalHeight": 1829,
"id": 106849034,
"title": "",
"description": "",
"sortOrder": 6,
"lastModified": "2023-05-25T10:39:44+02:00"
},
{
"url": "https://cis01.immoscout24.ch/urbaern---modern-living-bern?{width}x{height}/{resizemode}/{quality}/is24media/f8/ee/ef47ea8c40-230525.jpg",
"originalWidth": 3000,
"originalHeight": 2244,
"id": 106849035,
"title": "",
"description": "",
"sortOrder": 7,
"lastModified": "2023-05-25T10:40:04+02:00"
}
],
"lastPublished": "2023-10-19T13:27:02+02:00",
"sortalgoScore": 7.4925,
"sortalgoScore2": 2.331,
"listingCompletenessScore": 0,
"sortingPosition": 1,
"userRelevantScore": 0,
"isViewedProperty": false
}
]
Now that our immoscout24.ch scraper can successfully scrape search pages, let's scrape property pages.
Similar to hidden JSON data in search pages, there are also hidden JSON datasets in property listing pages:
To scrape immoscout24.ch property page data, we will extend our search scraping code with a scrape_properties() function:
import asyncio
import json
from typing import List, Dict
from httpx import AsyncClient, Response
from parsel import Selector
client = AsyncClient(
headers={
# use same headers as a popular web browser (Chrome on Windows in this case)
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Language": "en-US,en;q=0.9",
}
)
def parse_next_data(response: Response) -> Dict:
"""parse listing data from script tags"""
selector = Selector(response.text)
# extract data in JSON from the script tag
next_data = (
selector.xpath("//script[contains(text(),'INITIAL_STATE')]/text()").get().strip("window.__INITIAL_STATE__=")
)
# replace undefined values as Python doesn't recognize it
next_data = next_data.replace("undefined", "null")
if not next_data:
return
next_data_json = json.loads(next_data)
return next_data_json
async def scrape_properties(urls: List[str]) -> List[Dict]:
"""scrape listing data from immoscout24 proeprty pages"""
# add the property pages to a scraping list
to_scrape = [client.get(url) for url in urls]
properties = []
# scrape all property pages concurrently
for response in asyncio.as_completed(to_scrape):
data = parse_next_data(await response)
# handle expired property pages
try:
properties.append(data["listing"]["listing"])
except:
print("expired property page")
pass
print(f"scraped {len(properties)} property pages")
return properties
import asyncio
import json
from typing import List, Dict
from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse
scrapfly = ScrapflyClient(key="Your ScrapFly API key")
def parse_next_data(response: ScrapeApiResponse) -> Dict:
"""parse listing data from script tags"""
selector = response.selector
# extract data in JSON from the script tag
next_data = selector.xpath("//script[contains(text(),'INITIAL_STATE')]/text()").get().strip("window.__INITIAL_STATE__=")
# replace undefined values as Python doesn't recognize it
next_data = next_data.replace("undefined", "null")
if not next_data:
return
next_data_json = json.loads(next_data)
return next_data_json
async def scrape_properties(urls: List[str]) -> List[Dict]:
"""scrape listing data from immoscout24 proeprty pages"""
# add the property pages in a scraping list
to_scrape = [ScrapeConfig(url, asp=True, country="CH") for url in urls]
properties = []
# scrape all property pages concurrently
async for response in scrapfly.concurrent_scrape(to_scrape):
data = parse_next_data(response)
# handle expired property pages
try:
properties.append(data["listing"]["listing"])
except:
print("expired property page")
pass
print(f"scraped {len(properties)} property pages")
return properties
if __name__ == "__main__":
properties = asyncio.run(scrape_properties(
urls = [
"https://www.immoscout24.ch/rent/4000849390",
"https://www.immoscout24.ch/rent/4001086289",
"https://www.immoscout24.ch/rent/4001049042",
"https://www.immoscout24.ch/rent/4001079580"
]
))
# print the result in JSON format
print(json.dumps(properties, indent=2))
We use the parse_next_data function we used earlier to extract script tag data in JSON. Next, we add all the property page URLs to a scraping list and scrape them concurrently.
The result is a list containing all property page data:
{
"localization": {
"de": {
"text": {
"title": "Dachwohnung mit Charme an TOP-Lage in der Länggasse",
"description": "ruhige und zentral gelegene Liegenschaft - diverse Einkaufsmöglichkeiten sind in wenigen Gehminuten zu erreichen - die Universität sowie die öffentlichen Verkehrsmittel befinden sich in unmittelbarer Nähe<br />Wir vermieten per sofort oder nach Vereinbarung eine 4- Zimmer-Dachwohnung mit direktem Liftzugang in der Wohnung.<br /><br /> - schöne Parkettböden im allen Zimmern <br /> - Küche mit grossem Kühlschrank, Geschirrspüler und Bartheke<br /> - grosses Badezimmer mit Badewanne, Dusche und Doppellavabo<br /> - sep. WC mit Lavabo und Dachfenster<br /> - Wandschränke im Schlafzimmer und Korridor<br /> - Estrichzugang direkt aus der Wohnung<br /> - ein Kellerabteil gehört zur Wohnung dazu<br /> - kein Balkon vorhanden<br />"
},
"attachments": [
{
"caption": "image/jpeg",
"description": "Liegenschaft",
"file": "337f4f7440-231219.jpg",
"type": "IMAGE",
"url": "https://cdn.immoscout24.ch/listings/v2/f072/4000519178/image/cecdba83e399b270a7a6808669674dab.jpg"
},
{
"caption": "image/jpeg",
"description": "Küche",
"file": "ddd751cf48-231219.jpg",
"type": "IMAGE",
"url": "https://cdn.immoscout24.ch/listings/v2/f072/4000519178/image/5d17d17d9c717a9a40c6d05c84928974.jpg"
},
{
"caption": "image/jpeg",
"description": "Wohnzimmer mit Küchen-Bartheke",
"file": "b16d6c8147-231219.jpg",
"type": "IMAGE",
"url": "https://cdn.immoscout24.ch/listings/v2/f072/4000519178/image/57de586c8bb93ed6bb70f1cabc6c95b0.jpg"
},
{
"caption": "image/jpeg",
"description": "Wohnzimmer",
"file": "9293358a47-231219.jpg",
"type": "IMAGE",
"url": "https://cdn.immoscout24.ch/listings/v2/f072/4000519178/image/e140a112afb62cee9ff6357ca7990af5.jpg"
},
{
"caption": "image/jpeg",
"description": "Schlafzimmer",
"file": "0edbcdfe45-231219.jpg",
"type": "IMAGE",
"url": "https://cdn.immoscout24.ch/listings/v2/f072/4000519178/image/fa447bfe9565dd96593dacab3572bac7.jpg"
},
{
"caption": "image/jpeg",
"description": "Kinderzimmer",
"file": "f72664a645-231219.jpg",
"type": "IMAGE",
"url": "https://cdn.immoscout24.ch/listings/v2/f072/4000519178/image/d84d2ac035399538649c2741eaa61415.jpg"
},
{
"caption": "image/jpeg",
"description": "Bad",
"file": "8b8259dc41-231219.jpg",
"type": "IMAGE",
"url": "https://cdn.immoscout24.ch/listings/v2/f072/4000519178/image/883c66d83746fd43797c3074ad116edd.jpg"
},
{
"caption": "image/jpeg",
"description": "sep. WC",
"file": "9b512fcf4a-231219.jpg",
"type": "IMAGE",
"url": "https://cdn.immoscout24.ch/listings/v2/f072/4000519178/image/24bc38e97333f850c7e90e1b4109078c.jpg"
}
]
},
"primary": "de"
},
"lister": {
"legalName": "Robert Pfister AG",
"website": {
"value": "http://www.robertpfisterag.ch"
},
"address": {
"locality": "Bern",
"country": "CH",
"street": "Neuengasse 17",
"postalCode": "3011"
},
"externalPlatformId": "156142",
"immoscoutId": "156142",
"phone": "+41313203192",
"id": "f072",
"logoUrl": "https://media2.homegate.ch/t_customer_logo/logos/l_f072_v1.gif",
"contacts": {
"inquiry": {
"gender": "OTHER",
"phone": "+41313203192",
"familyName": "Vermietung"
},
"viewing": {
"gender": "OTHER",
"phone": "+41799638974",
"familyName": "Herr H. Rötheli"
}
}
},
"characteristics": {
"isCornerHouse": false,
"hasRamp": false,
"hasFireplace": false,
"numberOfRooms": 4,
"hasPowerSupply": false,
"isMiddleHouse": false,
"yearBuilt": 1920,
"isMinergieGeneral": false,
"isWheelchairAccessible": false,
"hasGardenShed": false,
"hasSwimmingPool": false,
"isOldBuilding": false,
"arePetsAllowed": false,
"floor": 3,
"hasGarage": false,
"hasFlatSharingCommunity": false,
"hasSewageSupply": false,
"hasParking": false,
"hasLiftingPlatform": false,
"hasBalcony": false,
"livingSpace": 92,
"hasWaterSupply": false,
"hasBuildingLawRestrictions": false,
"hasCableTv": false,
"isGroundFloorRaised": false,
"hasNiceView": false,
"hasConnectedBuildingLand": false,
"isUnderRoof": false,
"hasElevator": true,
"isNewBuilding": false,
"isMinergieCertified": false,
"hasGasSupply": false,
"isChildFriendly": false
},
"address": {
"country": "CH",
"geoDistances": [
{
"distance": -9235.730359934063,
"geoTag": "geo-canton-bern"
},
{
"distance": -1887.7265520081799,
"geoTag": "geo-city-bern"
},
{
"distance": -24.408342109690093,
"geoTag": "geo-citydistrict-stadtbach"
},
{
"distance": -24.408342109690093,
"geoTag": "geo-citydistrict-stadtbach-bern"
},
{
"distance": -359.12965408244463,
"geoTag": "geo-cityregion-laenggasse-felsenau-bern"
},
{
"distance": -359.12965408244463,
"geoTag": "geo-cityregion-langgasse-felsenau"
},
{
"distance": -48419.51198581247,
"geoTag": "geo-country-switzerland"
},
{
"distance": -7981.351123129005,
"geoTag": "geo-district-bern-mittelland"
},
{
"distance": -1904.5795177680163,
"geoTag": "geo-municipality-bern"
},
{
"distance": -1904.5795177680163,
"geoTag": "geo-region-bern"
},
{
"distance": -7981.351123129005,
"geoTag": "geo-region-bern-mittelland"
},
{
"distance": -1887.7265520081799,
"geoTag": "geo-zipcode-3000"
},
{
"distance": -335.32719461699145,
"geoTag": "geo-zipcode-3012"
}
],
"geoTags": [
"geo-canton-bern",
"geo-city-bern",
"geo-citydistrict-stadtbach",
"geo-citydistrict-stadtbach-bern",
"geo-cityregion-laenggasse-felsenau-bern",
"geo-cityregion-langgasse-felsenau",
"geo-country-switzerland",
"geo-district-bern-mittelland",
"geo-municipality-bern",
"geo-region-bern",
"geo-region-bern-mittelland",
"geo-zipcode-3000",
"geo-zipcode-3012"
],
"street": "Bühlstrasse 39",
"postalCode": "3012",
"locality": "Bern",
"geoCoordinates": {
"accuracy": "HIGH",
"manual": true,
"latitude": 46.951331561922,
"longitude": 7.430521732291
},
"region": "BE"
},
"externalIds": {
"internalReferenceId": "156142#idx-101081-430010--7181424##__f072",
"refHouse": "101081",
"refObject": "430010",
"displayPropertyReferenceId": "__f072.idx-101081-430010--7181424",
"platformListingId": "7181424",
"propertyReferenceId": "156142#idx-101081-430010--7181424##__f072"
},
"contactForm": {
"size": "NO_ADDRESS",
"deliveryFormat": "NORMAL"
},
"availableFrom": "2023-12-15",
"version": 5,
"platforms": [
"immoscout24"
],
"offerType": "RENT",
"meta": {
"createdAt": "2023-12-19T17:13:39.839Z",
"updatedAt": "2024-01-17T05:31:20.643Z",
"source": "IMMOHUB"
},
"publishers": [
{
"options": [
{
"value": "{\"topListingRanking\":0}",
"key": "top-listing"
},
{
"value": "{\"pubStartDate\":\"2023-12-19T00:00:00+01:00\",\"onlineStartDate\":\"2023-12-19T18:12:25+01:00\"}",
"key": "publication"
},
{
"value": "6.66000",
"key": "sorting-score"
},
{
"value": "OF",
"key": "object-source"
},
{
"value": "101081",
"key": "idx-ref-house"
},
{
"key": "idx-ref-prop"
},
{
"value": "430010",
"key": "idx-ref-object"
},
{
"value": "Unbekannt",
"key": "of-dispo-member-name"
},
{
"value": "1",
"key": "idx-type"
},
{
"value": "APPT",
"key": "idx-category"
},
{
"value": "M",
"key": "account-type"
},
{
"value": "156142",
"key": "account-id"
},
{
"key": "tags"
},
{
"value": "50725356",
"key": "last-synchronization-id"
}
],
"id": "is24-legacy-data"
}
],
"id": "4000519178",
"categories": [
"APARTMENT"
],
"prices": {
"rent": {
"area": "ALL",
"interval": "MONTH",
"gross": 2440
},
"currency": "CHF"
}
}
With this addition, our immoscout24.ch scraper can successfully scrape property listings from search and property pages. However, we can scrape it using the immoscout24.ch API. Let's take a look!
Currently, there is no public API available for immoscout24.ch. However, we can use the hidden API that is used by Immoscout24 front-end by reverse engineering and replicating the behavior of the browser in our scraper.
Learn how to find hidden APIs, how to scrape them, and what are some common challenges people face when developing web scrapers for hidden APIs.
To view this API, go to any search page on immoscout24.ch and open up theb browser developer tools. Then, go over the Network tab and filter by Fetch/XHR requests. Next, select the next search page which will display all the requests sent by the browser to the immoscout24.ch API:
On the left side, we can see all the XHR requests sent by the browser to the web server. On the right side, we can see the response and the headers sent along each request.
To get this data directly, we'll can replicate this API use in our Python web scraper:
import asyncio
import json
from typing import List, Dict
from httpx import AsyncClient
async def scrape_api(page_number: int) -> List[Dict]:
"""send a GET request to the immoscout24.ch API"""
client = AsyncClient(
headers={
# request header that we got from request header in developer tools
"authority": "rest-api.immoscout24.ch",
"is24-meta-pagenumber": str(page_number),
"is24-meta-pagesize": "24",
"origin": "https://www.immoscout24.ch",
"referer": "https://www.immoscout24.ch/",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Language": "en-US,en;q=0.9",
}
)
api_url = "https://rest-api.immoscout24.ch/v4/en/properties?l=436&s=1&t=1&inp=1"
data = await client.get(api_url)
return json.loads(data.text)
async def crawl_api(start_page: int, end_page: int, scrape_all_pages: bool) -> List[Dict]:
"""crawl the API by changing the page number in each request"""
first_page = await scrape_api(1)
max_search_pages = first_page["pagingInfo"]["totalPages"]
result = []
# scrape all pages if scrape_all_pages = True or end_page > max_search_pages
if scrape_all_pages == False and end_page <= max_search_pages:
end_page = end_page
else:
end_page = max_search_pages
# scrape the desired API pages
for page_number in range(start_page, end_page + 1):
data = await scrape_api(page_number)
result.extend(data["properties"])
return result
import asyncio
import json
from typing import List, Dict
from scrapfly import ScrapeConfig, ScrapflyClient
scrapfly = ScrapflyClient(key="Your ScrapFly API key")
async def scrape_api(page_number: int) -> List[Dict]:
"""send a GET request to the immoscout24.ch API"""
HEADERS = {
# request header that we got from request header in developer tools
"authority": "rest-api.immoscout24.ch",
"is24-meta-pagenumber": str(page_number),
"is24-meta-pagesize": "24",
"origin": "https://www.immoscout24.ch",
"referer": "https://www.immoscout24.ch/",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Language": "en-US,en;q=0.9",
}
api_url = "https://rest-api.immoscout24.ch/v4/en/properties?l=436&s=1&t=1&inp=1"
data = await scrapfly.async_scrape(
ScrapeConfig(url=api_url, asp=True, country="CH", headers=HEADERS)
)
return json.loads(data.scrape_result["content"])
async def crawl_api(start_page: int, end_page: int, scrape_all_pages: bool) -> List[Dict]:
"""crawl the API by changing the page number in each request"""
first_page = await scrape_api(1)
max_search_pages = first_page["pagingInfo"]["totalPages"]
result = []
# scrape all pages if scrape_all_pages = True or end_page > max_search_pages
if scrape_all_pages == False and end_page <= max_search_pages:
end_page = end_page
else:
end_page = max_search_pages
# scrape the desired API pages
for page_number in range(start_page, end_page + 1):
data = await scrape_api(page_number)
result.extend(data["properties"])
return result
if __name__ == "__main__":
result = asyncio.run(crawl_api(start_page=1, end_page=10, scrape_all_pages=False))
# print the result in JSON format
print(json.dumps(result, indent=2))
with open ("api-data.json", "r", encoding="utf-8") as file:
json.dump(result, file, indent=2)
The result is the same as the result we got earlier from scraping search pages:
[
{
"id": 7395340,
"accountId": 49638,
"companyId": 27042,
"memberPackageId": 2,
"agency": {
"companyCity": "Bern",
"companyName1": "U.C. Buchschacher AG",
"companyPhoneMobile": "031 311 71 26",
"companyStreet": "Grabenpromenade 5",
"companyZip": "3011",
"showLogoOnSerp": false,
"logoUrl": "https://cis01.immoscout24.ch/memberlogos/L2327-R.png",
"logoUrlDetailPage": "https://cis01.immoscout24.ch/memberlogos/L2327-R.png",
"isAccountMigrated": true,
"isGuest": false,
"userType": "M"
},
"availableFrom": "2024-02-01T00:00:00+01:00",
"availableFromFormatted": "01.02.2024",
"cityId": 295,
"cityName": "Bern",
"commuteTimes": {
"defaultPois": [
{
"defaultPoiId": 2,
"label": "Bern station",
"transportations": [
{
"transportationTypeId": 6,
"travelTime": 623,
"isReachable": true
},
{
"transportationTypeId": 10,
"travelTime": 775,
"isReachable": true
},
{
"transportationTypeId": 3,
"travelTime": 541,
"isReachable": true
}
]
}
]
},
"countryId": 1,
"extraPrice": 200,
"extraPriceFormatted": "CHF 200.\u2014",
"geoAccuracy": 8,
"grossPrice": 2030,
"grossPriceFormatted": "CHF 2030.\u2014",
"hasNewBuildingProject": false,
"hasVirtualTour": false,
"isHighlighted": false,
"isNeubauLite": true,
"isNeubauLitePremium": true,
"isHgCrosslisting": false,
"isNew": false,
"isNewEndDate": "2023-10-21T10:36:08+02:00",
"isOnline": true,
"isTopListing": false,
"isPremiumToplisting": true,
"lastModified": "2023-11-07T06:46:24+01:00",
"latitude": 46.954821852012,
"longitude": 7.458991284352,
"netPrice": 1830,
"netPriceFormatted": "CHF 1830.\u2014",
"normalizedPrice": 2030,
"normalizedPriceFormatted": "CHF 2030.\u2014",
"numberOfRooms": 2.5,
"numberOfRoomsFormatted": "2.5 rooms",
"offerTypeId": 1,
"priceUnitId": 6,
"priceUnitLabel": "month",
"price": 2030,
"priceFormatted": "CHF 2030.\u2014",
"propertyDetailUrl": "/en/d/flat-rent-bern/7395340?s=1&t=1&l=436&inp=1&ct=506&ci=1&pn=1",
"propertyUrl": "/en/d/flat-rent-bern/7395340",
"searchParameters": {
"s": "1",
"t": "1",
"l": "436",
"inp": "1",
"ct": "506",
"ci": "1",
"pn": "1"
},
"propertyCategoryId": 1,
"propertyTypeId": 1,
"stateShort": "BE",
"state": "Bern",
"street": "Funkerstrasse 25",
"surfaceLiving": 80,
"surfaceLivingFormatted": "80 m\u00b2",
"title": "2.5-Zimmerwohnung im \"Breitsch-Quartier\"",
"videoViewingEnabled": false,
"zip": "3013",
"zipId": 863,
"packageType": 0,
"shortDescription": "In renoviertem MFH vermieten wir eine grossfl\u00e4chige und helle 2.5-Zimmerwohnung.Nebst dem grossz\u00fcgigen Grundriss, verf\u00fcgt die Wohnung \u00fcber folgenden Ausbaustandard:Eichenparkett- und Plattenbodenmoderne K\u00fcche mit hochwertigen V-Zug Ger\u00e4tenBad/WC mit moderner Ausstattunggedeckter BalkonReduit in der WohnungKellerabteil, LiftEinstellhallenparkplatz kann dazugemietet werdenIn unmittelbarer N\u00e4he befinden sich zahlreiche Verpflegungs- und Einkaufsm\u00f6glichkeiten, sowie liegt die Bushaltestelle fast vor der Haust\u00fcr.Vereinbaren Sie noch heute\u00a0Ihren Besi",
"listingLayout": 3,
"packageScore": 0,
"isQualityListing": true,
"images": [
{
"url": "https://cis01.immoscout24.ch/2-5-zimmerwohnung-im-breitsch-quartier--bern?{width}x{height}/{resizemode}/{quality}/is24media/0a/8d/9509679c41-221104.jpg",
"originalWidth": 0,
"originalHeight": 0,
"id": 100551937,
"title": "",
"description": "",
"sortOrder": 1,
"lastModified": "2022-11-04T13:42:08+01:00"
},
{
"url": "https://cis01.immoscout24.ch/2-5-zimmerwohnung-im-breitsch-quartier--bern?{width}x{height}/{resizemode}/{quality}/is24media/07/2f/902764364f-221104.JPG",
"originalWidth": 0,
"originalHeight": 0,
"id": 100551938,
"title": "",
"description": "",
"sortOrder": 2,
"lastModified": "2022-11-04T13:42:08+01:00"
},
{
"url": "https://cis01.immoscout24.ch/2-5-zimmerwohnung-im-breitsch-quartier--bern?{width}x{height}/{resizemode}/{quality}/is24media/bc/61/b3aac22747-221104.jpg",
"originalWidth": 0,
"originalHeight": 0,
"id": 100551939,
"title": "",
"description": "",
"sortOrder": 3,
"lastModified": "2022-11-04T13:42:08+01:00"
},
{
"url": "https://cis01.immoscout24.ch/2-5-zimmerwohnung-im-breitsch-quartier--bern?{width}x{height}/{resizemode}/{quality}/is24media/82/88/da9df5cb48-221104.jpg",
"originalWidth": 0,
"originalHeight": 0,
"id": 100551940,
"title": "",
"description": "",
"sortOrder": 4,
"lastModified": "2022-11-04T13:42:08+01:00"
},
{
"url": "https://cis01.immoscout24.ch/2-5-zimmerwohnung-im-breitsch-quartier--bern?{width}x{height}/{resizemode}/{quality}/is24media/d2/54/54285c8343-221104.jpg",
"originalWidth": 0,
"originalHeight": 0,
"id": 100551941,
"title": "",
"description": "",
"sortOrder": 5,
"lastModified": "2022-11-04T13:42:08+01:00"
},
{
"url": "https://cis01.immoscout24.ch/2-5-zimmerwohnung-im-breitsch-quartier--bern?{width}x{height}/{resizemode}/{quality}/is24media/da/19/9c108e8a4d-221104.jpg",
"originalWidth": 0,
"originalHeight": 0,
"id": 100551942,
"title": "",
"description": "",
"sortOrder": 6,
"lastModified": "2022-11-04T13:42:08+01:00"
},
{
"url": "https://cis01.immoscout24.ch/2-5-zimmerwohnung-im-breitsch-quartier--bern?{width}x{height}/{resizemode}/{quality}/is24media/89/04/739322b449-221104.jpg",
"originalWidth": 0,
"originalHeight": 0,
"id": 100551943,
"title": "",
"description": "",
"sortOrder": 7,
"lastModified": "2022-11-04T13:42:09+01:00"
},
{
"url": "https://cis01.immoscout24.ch/2-5-zimmerwohnung-im-breitsch-quartier--bern?{width}x{height}/{resizemode}/{quality}/is24media/3e/33/bfcd9ea34a-221104.jpg",
"originalWidth": 0,
"originalHeight": 0,
"id": 100551944,
"title": "",
"description": "",
"sortOrder": 8,
"lastModified": "2022-11-04T13:42:09+01:00"
},
{
"url": "https://cis01.immoscout24.ch/2-5-zimmerwohnung-im-breitsch-quartier--bern?{width}x{height}/{resizemode}/{quality}/is24media/76/fb/ef180d4d4a-221104.jpg",
"originalWidth": 0,
"originalHeight": 0,
"id": 100551945,
"title": "",
"description": "",
"sortOrder": 9,
"lastModified": "2022-11-04T13:42:09+01:00"
},
{
"url": "https://cis01.immoscout24.ch/2-5-zimmerwohnung-im-breitsch-quartier--bern?{width}x{height}/{resizemode}/{quality}/is24media/8d/d1/8c7deb7d4d-221104.jpg",
"originalWidth": 0,
"originalHeight": 0,
"id": 100551946,
"title": "",
"description": "",
"sortOrder": 10,
"lastModified": "2022-11-04T13:42:09+01:00"
}
],
"lastPublished": "2023-10-19T10:36:12+02:00",
"sortalgoScore": 5.8275,
"sortalgoScore2": 2.331,
"listingCompletenessScore": 0,
"sortingPosition": 1,
"userRelevantScore": 0,
"isViewedProperty": false
}
]
With this addition, we can scrape immoscout24.ch to get property listing data in two different ways ensuring that our scrapers are always on top of the latest property listings in Switzerland.
However, the chances of getting our scraper blocked after sending additional requests are high. Let's take a look at a solution!
To avoid immoscout24.ch web scraping blocking, we'll use ScrapFly API - a web scraping API that powers up web scrapers for scraping at scale with:
By using the ScrapFly asp feature with the ScrapFly Python SDK, we can scrape any amount of data from immoscout24.ch without getting blocked:
import httpx
from parsel import Selector
response = httpx.get("some immoscout24.ch url")
selector = Selector(response.text)
# in ScrapFly SDK becomes
from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse
scrapfly_client = ScrapflyClient("Your ScrapFly API key")
result: ScrapeApiResponse = scrapfly_client.scrape(ScrapeConfig(
# some homegate.ch URL
"https://www.immoscout24.ch/en/d/flat-rent-bern/8068164",
# we can select specific proxy country
country="CH",
# and enable anti scraping protection bypass:
asp=True,
# allows JavaScript rendering similar to headless browsers
render_js=True
))
# use the built-in parsel selector
selector = result.selector
To wrap up this guide on scraping immoscout24.ch, let's take a look at some frequently asked questions.
Since immoscout24.ch data is publicly available - it's perfectly legal to scrape as long as scrapers don't directly damage the website. Note that using private scraped seller's data can be difficult due to GDPR in EU and for more information, refer to our is web scraping legal? page.
Currently, there is no public API for immoscout24.ch. However, Immoscout can be scraped with Python through multiple publicly available endpoints.
Yes, for more real estate property scraping targets see #realestate scraping tag
In this guide, we explained how to scrape immoscout24.ch - a popular real estate listing website - using nothing but a bit of Python.
We went through a step-by-step guide on scraping immoscout24.ch property and search pages. Furthermore, we explored how to use immoscout24.ch private API to get property listing data and how to avoid immoscout.ch web scraper blocking.
In this article, we'll explore how to scrape Reddit. We'll extract various social data types from subreddits, posts, and user pages. All of which through plain HTTP requests without headless browser usage.
In this scrape guide we'll be taking a look at one of the most popular web scraping targets - LinkedIn.com. We'll be scraping people profiles, company profiles as well as job listings and search.
In this guide, we'll explain how to scrape SimilarWeb through a step-by-step guide. We'll scrape comprehensive website traffic insights, websites comparing data, sitemaps, and trending industry domains.