Immowelt.de is a popular website for real estate ads in Germany, featuring thousands of real estate property ads.
In this article, we'll cover how to scrape real estate data from immowelt.de property pages and search API. We'll also explain how to avoid immowelt.de web scraping blocking. Let's dive in!
This tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect and here's a good summary of what not to do:
Scrapfly does not offer legal advice but these are good general rules to follow in web scraping
and for more you should consult a lawyer.
Exploring thousands of property listings on Immowelt.de manually can be time-consuming. However, web scraping can extract loads of data with just a few code lines, saving a lot of time and resources.
Scrapping immowelt.de enables investors and buyers to explore potential investment opportunities by discovering low-priced properties with high growth potential.
Immowelt.de web scraping can help real estate businesses monitor competitors' listings and pricing strategies, allowing for remaining up-to-date with the market changes and staying competitive.
In this guide on web scraping immowelt.de, we'll use a few Python libraries:
httpx: For sending HTTP requests to the website.parsel: For parsing the HTML using XPath and CSS selectors.asyncio: For runnA Python SDK for ScrapFly, a web scraping API that allows for scraping at scale and bypassing any web scraping blocking.scrapfly-sdk: A Python SDK for ScrapFly, a web scraping API that allows for scraping at scale and bypassing any web scraping blocking.You don't need to install asyncio, as it comes pre-installed in Python. Install the other libraries using the following pip command:
pip install httpx parsel scrapfly-sdk
Let's begin by scraping immowelt.de property pages. Go to any property page on the website and you will get a page similar to this:
Instead of scraping this page content using selectors like CSS and XPath, we'll extract the data directly in JSON from script tags. To see this data, open browser developer tools and scroll down to the script tag with the id serverApp-state:
The data inside this tag is the same data seen on the web page but before getting rendered into the HTML. Which is often known as hidden web data. However, this data isn't a valid JSON yet as it's encoded with &q; characters. To scrape it, we'll select this script tag and decode the data inside:
import asyncio
import json
from typing import Dict, List
from httpx import AsyncClient, Response
from parsel import Selector
client = AsyncClient(
headers={
# use same headers as a popular web browser (Chrome on Windows in this case)
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Language": "en-US,en;q=0.9",
}
)
def parse_next_data(response: Response) -> Dict:
"""parse lsiting data from property page"""
# get the property id from the page URL
proeprty_id = str(response.url).split(".de/")[-1]
selector = Selector(response.text)
# extract the data from the script tag
next_data = selector.xpath("//script[@id='serverApp-state']/text()").get()
# parse the data into a valid JSON object
next_data = json.loads(next_data.replace("&q;", '"').replace("\n", " "))
return next_data[proeprty_id]
async def scrape_properties(urls: List[str]) -> List[Dict]:
"""scrape listings data from property pages"""
# add all property pages to a scraping list
to_scrape = [client.get(url) for url in urls]
properties = []
# scrape all property pages concurrently
for response in asyncio.as_completed(to_scrape):
try:
data = parse_next_data(await response)
properties.append(data)
except:
print("expired property page")
print(f"scraped {len(properties)} property pages")
return properties
import asyncio
import json
from typing import Dict, List
from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse
scrapfly = ScrapflyClient(key="Your ScrapFly API key")
def parse_next_data(response: ScrapeApiResponse) -> Dict:
"""parse lsiting data from property page"""
# get the property id from the page URL
proeprty_id = str(response.context["url"]).split(".de/")[-1]
selector = response.selector
# extract the data from the script tag
next_data = selector.xpath("//script[@id='serverApp-state']/text()").get()
# parse the data into a valid JSON object
next_data = json.loads(next_data.replace("&q;", '"').replace("\n", " "))
return next_data[proeprty_id]
async def scrape_properties(urls: List[str]) -> List[Dict]:
"""scrape listings data from property pages"""
# add all property pages to a scraping list
to_scrape = [ScrapeConfig(url, asp=True, country="DE") for url in urls]
properties = []
# scrape all property pages concurrently
async for response in scrapfly.concurrent_scrape(to_scrape):
try:
data = parse_next_data(response)
properties.append(data)
except:
print("expired property page")
print(f"scraped {len(properties)} property pages")
return properties
if __name__ == "__main__":
data = asyncio.run(scrape_properties(
urls = [
"https://www.immowelt.de/expose/27t9c5f",
"https://www.immowelt.de/expose/27dgc5f",
"https://www.immowelt.de/expose/25jqw5t",
"https://www.immowelt.de/expose/249p65w",
"https://www.immowelt.de/expose/24c7p5k"
]
))
# print the data in JSON format
print(json.dumps(data, indent=2))
We start by creating an async httpx client with standard web browser headers. We then define our scraping functions:
parse_next_data() function to select and parse the hidden web data into a valid JSON object.scrape_properties() function to scrape the property pages by adding their URLs to a scraping list and concurrently scraping them.The result is a list containing all the property page data:
[
{
"GlobalObjectKey": "01c0c334-fd75-442a-bc86-83894df68d0a",
"CreateDate": "2023-10-23T12:09:08.723Z",
"General": {
"Headline": "Moderne, hochwertige Eigentumswohnung mit Grundwasserw\u00e4rmepumpe und Photovoltaikanlage",
"EstateType": "APARTMENT",
"EstateTypeKey": "WOHNUNG",
"ReferenceNumber": "CB.2309",
"DistributionTypeKey": "ZUM_KAUF",
"DistributionType": "BUY",
"Rooms": 2,
"LivingSpace": 54.7
},
"Seo": {
"BreadCrumb": [
{
"Id": "https://www.immowelt.de/",
"Name": "Immobilien"
},
{
"Id": "https://www.immowelt.de/suche/wohnungen",
"Name": "Wohnungssuche"
},
{
"Id": "https://www.immowelt.de/suche/wohnungen/kaufen",
"Name": "Kaufen"
},
{
"Id": "https://www.immowelt.de/suche/muenchen-allach-untermenzing/wohnungen/kaufen",
"Name": "M\u00fcnchen (Allach-Untermenzing)"
},
{
"Id": "https://www.immowelt.de/liste/muenchen-allach-untermenzing/wohnungen/kaufen?sort=relevanz",
"Name": "Ergebnisliste"
}
],
"MetaTags": {
"PageTitle": "Moderne, hochwertige Eigentumswohnung mit Grundwasserw\u00e4rmepumpe und Photovoltaikanlage | Wohnung M\u00fcnchen",
"Description": "Wohnung in M\u00fcnchen zum Kauf mit 2 Zimmer und 54,7 m\u00b2 Wohnfl\u00e4che. Ausstattung: barrierefrei, Kelleranteil, Personenaufzug, Erstbezug, Fu\u00dfbodenheizung, Zentralheizung."
},
"SeoOptimizedLocation": "muenchen-allach-untermenzing"
},
"Tealium": {
"enh_promo_id": [
"kfw",
"interhyp",
"treppenmeister"
],
"enh_promo_name": [
"kfw",
"interhyp_text",
"treppenmeister"
],
"enh_promo_creative": [
"expose_teaser",
"expose_servicebereich",
"expose_teaser"
],
"enh_promo_position": [
"expose",
"expose",
"expose"
],
"broker_guid": "56453",
"object_count_photos": 7,
"object_state": [
"Deutschland"
],
"object_locationid": "464951",
"object_city": [
"M\u00fcnchen"
],
"object_district": [
"Allach-Untermenzing"
],
"object_federalstate": [
"Bayern"
],
"object_county": [
"M\u00fcnchen"
],
"object_zip": "80999",
"object_address_is_visible": false,
"object_price": 599000,
"object_currency": "EUR",
"object_gok": [
"01c0c334-fd75-442a-bc86-83894df68d0a"
],
"object_features": [
"barriefrei",
"Kelleranteil",
"Personenaufzug",
"ERSTBEZUG",
"Fu\u00dfbodenheizung",
"Zentralheizung",
"Tiefgarage",
"wasch_trockenraum",
"kable_sat_tv"
],
"object_rooms": 2,
"object_area": 54.7,
"object_marketingtype": [
"KAUF"
],
"object_objektart": [
"wohnung"
],
"object_objektart_sub": "wohnung",
"product_id": [
"500"
],
"product_name": [
"kontakt"
],
"product_price": [
0
],
"product_price_total": 0,
"product_variant": [
""
],
"product_quantity": [
1
],
"product_period": [
"0"
],
"product_payment": [
""
],
"product_coupon": "",
"product_discount": [
0
],
"product_net_price": [
0
],
"product_net_price_total": "0.00",
"product_business_type": "living",
"product_building_state": "FIRST_TIME_USE",
"expose_type": "project",
"page_type": "productdetailpage",
"app_medienid": "immowelt.de"
},
"InternalAds": [
{
"Id": "TREPPENMEISTER",
"URL": "https://www.immowelt.de/customer/treppenmeister/"
}
],
"Advertisement": {
"TargetValues": {
"IMMO_ADGROUP": "IMMOBILIEN_EXPOSE",
"IMMO_ART": 1,
"IMMO_ESTATETYPE": "APARTMENT",
"IMMO_MK": 1,
"IMMO_DISTRIBUTION": "BUY",
"IMMO_NT": 13,
"IMMO_P": "599000",
"IMMO_PRANGE": "500000-1000000",
"IMMO_GLOBALUSERID": 56453,
"IMMO_LOCATIONID": 464951,
"IMMO_ORTCLEAN": "M\u00fcnchen",
"IMMO_ORT": "M\u00fcnchen (Allach-Untermenzing)",
"IMMO_PLZ": "80999",
"IMMO_TITLE": "Moderne, hochwertige Eigentumswohnung mit Grundwasserw\u00e4rmepumpe und Photovoltaikanlage",
"IMMO_BILD": "https://ms.immowelt.org/56094624-23c5-4ded-9aaa-a3b74850a63e/e273898e-9cf6-41d1-98b0-0595f86b50b6",
"IMMO_ZA": 2,
"IMMO_WFL": 54.7
},
"CampaignLinks": [
"UMZUG",
"ODA",
"KFW"
],
"IsAllowedForFinancing": true
},
"EquipmentAreas": [
{
"Key": "APARTMENT",
"Headline": "Die Wohnung",
"Equipments": [
{
"Key": "FLOOR",
"Value": "3. Geschoss",
"DisplayType": "TABLE",
"Label": "Wohnungslage"
},
{
"Key": "SUITABILITY",
"Value": "barrierefrei",
"DisplayType": "LIST",
"Label": ""
},
{
"Key": "CONDITION",
"Value": "Erstbezug",
"DisplayType": "LIST",
"Label": "Zustand"
},
{
"Key": "OTHERROOMS",
"Value": "Kelleranteil, Wasch-Trockenraum",
"DisplayType": "LIST",
"Label": "Weitere R\u00e4ume"
},
{
"Key": "CONNECTIVITY",
"Value": "Kabelanschluss",
"DisplayType": "LIST",
"Label": "Anschl\u00fcsse"
}
]
},
{
"Key": "SECONDARY",
"Headline": "Wohnanlage",
"Equipments": [
{
"Key": "ELEVATOR",
"Value": "Personenaufzug",
"DisplayType": "LIST",
"Label": ""
},
{
"Key": "PARKINGSLOT",
"Value": "Tiefgarage",
"DisplayType": "LIST",
"Label": "1 Stellplatz"
}
]
},
{
"Key": "ENERGY",
"Headline": "Energie / Versorgung",
"Equipments": [
{
"Key": "HEATING",
"Value": "Fu\u00dfbodenheizung, Zentralheizung",
"DisplayType": "LIST",
"Label": "Heizungsart"
}
]
}
],
"HardFacts": [
{
"NumberValue": 599000,
"Unit": "EUR",
"Key": "PRICE",
"Label": "Kaufpreis"
},
{
"Key": "AREA_LIVING",
"Label": "Wohnfl\u00e4che",
"NumberValue": 54.7,
"Unit": "SQM",
"Comments": [
"ca."
]
},
{
"Key": "ROOMS",
"Label": "Zimmer",
"NumberValue": 2
}
],
"MetaBadges": [],
"LocalRatings": {
"lastUpdated": "2023-11-06T17:05:22.000Z",
"scores": {
"local_amenities": 1.6,
"mobility": 7.4
}
},
"Texts": [
{
"Title": "Lage",
"Content": "Untermenzing ist ein Stadtteil im M\u00fcnchner Westen und geh\u00f6rt ebenfalls zum Stadtbezirk Pasing-Obermenzing. Der Stadtteil liegt direkt am Ufer der W\u00fcrm und ist von viel Gr\u00fcn umgeben.\n\nUntermenzing ist vor allem durch seine ruhige und l\u00e4ndliche Lage bekannt. Der Stadtteil besteht haupts\u00e4chlich aus Einfamilienh\u00e4usern und kleinen Wohnanlagen. Die Stra\u00dfen von B\u00e4umen ges\u00e4umt, was dem Viertel eine idyllische Atmosph\u00e4re verleiht.\n\nDas Zentrum von Untermenzing bildet der historische Ortskern mit der Pfarrkirche St. Michael und dem alten Rathaus. Hier gibt es einige Gesch\u00e4fte, Restaurants und Caf\u00e9s, die zum Verweilen einladen. Auch der Wochenmarkt, der regelm\u00e4\u00dfig stattfindet, ist ein beliebter Treffpunkt f\u00fcr die Bewohner.\n\nNeben dem Ortskern bietet Untermenzing auch zahlreiche Gr\u00fcnfl\u00e4chen und Naherholungsgebiete. Der nahegelegene W\u00fcrmkanal und der Aubinger Lohe sind beliebte Ziele f\u00fcr Spazierg\u00e4nge und Radtouren. Auch der nahegelegene Langwieder See bietet M\u00f6glichkeiten f\u00fcr Freizeitaktivit\u00e4ten im Gr\u00fcnen.\n\nDie Verkehrsanbindung in Untermenzing ist gut. Mit der S-Bahn-Linie S2 ist man in etwa 20 Minuten am M\u00fcnchner Hauptbahnhof. Auch die Autobahn A8 ist schnell zu erreichen, sodass man schnell in Richtung Augsburg oder Stuttgart fahren kann.\n\nInsgesamt ist Untermenzing ein ruhiger und naturnaher Stadtteil, der vor allem durch seine l\u00e4ndliche Atmosph\u00e4re und die gute Anbindung an die Innenstadt \u00fcberzeugt.",
"Position": 1
},
{
"Title": "Objektbeschreibung",
"Content": "Die Neubauwohnung ist eine moderne und stilvolle Wohnung, die derzeit fertiggestellt wird. Sie befindet sich in einem Mehrfamilienhaus, das auf einem ca. 799 m\u00b2 gro\u00dfen Grundst\u00fcck steht. Sie bietet alle Annehmlichkeiten eines zeitgem\u00e4\u00dfen Wohnens.\n\nDie Wohnung verf\u00fcgt \u00fcber gro\u00dfz\u00fcgige und lichtdurchflutete R\u00e4ume, die mit hochwertigen Materialien, wie z. B. Parkettboden und Fu\u00dfbodenheizung ausgestattet ist. Der offene Grundriss schafft eine angenehme Wohnatmosph\u00e4re und erm\u00f6glicht eine flexible Gestaltung der Wohnbereiche.\n\nDas Badezimmer ist modern und elegant gestaltet und verf\u00fcgt \u00fcber eine ebenerdige Dusche mit Echtglas-Duschtrennwand.\n\nDie Wohnung befindet sich im Obergeschoss und verf\u00fcgt \u00fcber einen Balkon, auf dem man die frische Luft genie\u00dfen und den Ausblick auf die Umgebung genie\u00dfen kann. \n\nZur Ausstattung der Wohnung geh\u00f6ren auch ein Aufzug, ein Kellerabteil und ein Parkplatz in der Tiefgarage. Die Wohnung ist gut isoliert und verf\u00fcgt \u00fcber eine effiziente Heizungsanlage, um den Energieverbrauch zu minimieren.\n\nDie Neubauwohnung befindet sich in einer attraktiven Lage mit guter Anbindung an \u00f6ffentliche Verkehrsmittel und Einkaufsm\u00f6glichkeiten. Sie bietet eine moderne und komfortable Wohnl\u00f6sung f\u00fcr Singles, Paare oder Familien, die ein zeitgem\u00e4\u00dfes und stilvolles Zuhause suchen.\n\nDas Haus entsteht in massiver Bauweise und die Wohnung wird schl\u00fcsselfertig, inklusiver der Au\u00dfenanlagen und Bepflanzung \u00fcbergeben.\n\nNehmen Sie alle Annehmlichkeiten eines Neubaus in Anspruch - wie Zahlung nach Baufortschritt - neuste Technik - Gew\u00e4hrleistung!\n\nDie Einhaltung des hohen Qualit\u00e4tsanspruches, der an die einzelnen Handwerksbetriebe gestellt wird, wird vom T\u00dcV S\u00dcD in Form eines Baucontrollings \u00fcberwacht. Der Bautr\u00e4ger, die Forma Hausbau GmbH hat in der Umgebung von M\u00fcnchen bereits zahlreiche Bauprojekte verwirklicht.\n\nIn diesem Haus stehen noch weitere Wohnungen zum Kauf zur Verf\u00fcgung.\n\nGerne unterst\u00fctzen wir Sie auch bei der Auswahl einer Finanzierung.",
"Position": 2
},
{
"Title": "Ausstattung",
"Content": "Details der Ausstattung:\n\n- PV Anlage zur Stromerzeugung f\u00fcr W\u00e4rmepumpe \n einschl. Batteriespeicher\n- Ziegelbauweise mit W\u00e4rmed\u00e4mmung\n- Elektrische Rolll\u00e4den mit Zeitschaltuhr\n- Fu\u00dfbodenheizung in allen Wohn- und Hobbyr\u00e4umen\n- Eiche-Parkettb\u00f6den\n- 3-fach Isolierverglasung\n- Beheizung mit moderner W\u00e4rmepumpe der Firma Viessmann \noder gleichwertig.\n- Bodengleiche Duschen mit Echtglas-Duschtrennw\u00e4nden\n- Baubegleitendes T\u00dcV Baucontrolling mit Dokumentation\n- Dezentrales Be- und Entl\u00fcftungssystem \n mit W\u00e4rmer\u00fcckgewinnung\n- Umfangreiche Elektroausstattung mit CAT-7 EDV Verkabelung\n\nWeitere Informationen stellen wir Ihnen gerne anhand der Baubeschreibung zur Verf\u00fcgung.\n\nGerne stehen wir f\u00fcr ein pers\u00f6nliches Gespr\u00e4ch oder einen Besichtigungstermin zur Verf\u00fcgung - wir freuen uns auf Ihre Kontaktaufnahme!",
"Position": 3
},
{
"Title": "Preisinformation",
"Content": "1 Tiefgaragenstellplatz, Kaufpreis: 30.000,00 EUR\n",
"Position": 4
},
{
"Title": "Weitere Informationen",
"Content": "Sonstiges\nEs liegt ein Energiebedarfsausweis vor.\nDieser ist g\u00fcltig bis 9.5.2023.\nEndenergiebedarf betr\u00e4gt 10.40 kwh/(m\u00b2*a).\nWesentlicher Energietr\u00e4ger der Heizung ist Luft/wasser W\u00e4rmepumpe.\nDie Energieeffizienzklasse ist A+.\n\nSonstige Angaben:\nIhre Ansprechpartnerin:\nChristine Burmeister\n\nTel. 0160-1878608\n\nErlaubnis nach \u00a734c der Gewerbeordnung erteilt durch das Landratsamt Starnberg, Strandbadstr. 2, 82319 Starnberg.\n\nAlle Angaben sind ohne Gew\u00e4hr und basieren ausschlie\u00dflich auf Informationen, die uns vom Auftraggeber zur Verf\u00fcgung gestellt wurden. Wir \u00fcbernehmen keine Gew\u00e4hr f\u00fcr die Vollst\u00e4ndigkeit, Richtigkeit und Aktualit\u00e4t dieser Angaben. Die Nutzung bzw. Weitergabe s\u00e4mtlicher Bilder und grafischen Darstellungen ist nicht gestattet. Mit der Anfrage auf dieses Objekt erkl\u00e4rt der Interessent seine ausdr\u00fcckliche Einwilligung f\u00fcr die Kontaktaufnahme per Telefon oder e-Mail durch einen unserer Mitarbeiter.\n\nStichworte\nTiefgarage vorhanden, Nutzfl\u00e4che: 6,60 m\u00b2, vermietbare Fl\u00e4che: 54,70 m\u00b2, Anzahl der Schlafzimmer: 1, Anzahl der Badezimmer: 1, Anzahl Balkone und Terrassen: 1, Balkon-Terrassen-Fl\u00e4che: 6,40 m\u00b2, Kellerfl\u00e4che: 3,40 m\u00b2",
"Position": 5
}
],
"Price": {
"AdditionalInformation": {
"Commission": {
"CommissionType": "COMMISSION_DEFAULT",
"DisplayValue": {
"Label": "Provision f\u00fcr K\u00e4ufer",
"StringValue": "Bitte beachte, das Angebot kann bei Vertragsabschluss die Zahlung einer Provision beinhalten. Weitere Informationen erh\u00e4ltst Du vom Anbieter."
}
},
"MarketPricing": {
"Heading": "Wohnungspreise in M\u00fcnchen (Allach-Untermenzing)",
"Link": "https://www.immowelt.de/immobilienpreise/muenchen-allach-untermenzing/wohnungspreise"
}
},
"DataTable": [
{
"NumberValue": 599000,
"Unit": "EUR",
"Key": "PRICE_BUYING_PRICE",
"Label": "Kaufpreis"
},
{
"NumberValue": 30000,
"Unit": "EUR",
"Key": "PRICE_PARKINGPRICE",
"Label": "1 Stellplatz"
}
]
},
"EstateAddress": {
"Country": "Deutschland",
"City": "M\u00fcnchen",
"District": "Allach-Untermenzing",
"ZipCode": "80999",
"LocationId": 464951,
"PublishStreet": false,
"FederalState": "Bayern",
"FederalStateId": 2
},
"EstateMapData": {
"LocationCoordinates": {
"Latitude": 48.19266,
"Longitude": 11.46333
},
"ShowPin": false,
"LocationId": 464951,
"LocatedIn": "DE",
"Locations": [
{
"latitude": 48.19266,
"longitude": 11.46333,
"locationId": 464951,
"name": "Allach-Untermenzing",
"adminTypeName": "Stadtteil",
"locationTypeId": 10
},
{
"locationId": 4916,
"name": "M\u00fcnchen",
"adminTypeName": "Kreisfreie Stadt",
"locationTypeId": 6
},
{
"locationId": 115926,
"name": "Oberbayern",
"adminTypeName": "Regierungsbezirk",
"locationTypeId": 5
},
{
"locationId": 122156,
"name": "Bayern",
"adminTypeName": "Bundesland",
"locationTypeId": 4
},
{
"locationId": 121726,
"name": "Deutschland",
"adminTypeName": "Land",
"locationTypeId": 2
}
],
"Polygons": [
{
"polygon": "q}eeHsmaeAyKoKiS`DiEyHuI{@yAbAs@xBmBpJwAbQ}HiBaGqE{M~_@RdKrBzKhXlu@~CdEbEhD~C@hAl@dArEdJvC|CnBvIfhAv@pHbBxFj@dWrRvFxDMlThApNoAzH}DrWyDtKlMd@~]nEnHlKbFrHwCnHyLrLGdCmNjDgLvCeEzFeFvFcHbA{BdC}GaAgPxCsFzBi@wA}T{BuDiDaVkCoBcFe@aAaFhBwLnAkDrHkIf@yAc@sEn@aPgCsXgAyC_AkIl@mHZsAbB}@fK{OrLiWoGaL]kXd@gAQwC[SImLqCVmD|Bw@dD_EpDqUcAyAgAmAGiHFqKq@sCFuFaDqAj@uBfC}AVo@[e@ZcAo@yAFuGcBuGvC{CdCeDbH{@NkHuGaDc@eBJaJzMe@dWgChIR`K}FpN_DRcC_AqVaNPuK"
}
]
},
"MediaItems": [
{
"BaseUri": "https://ms.immowelt.org",
"FileName": "ec7ea2ab-4ea0-494b-babf-8da4a08e3fe9__94193.jpg",
"FileSize": 223954,
"Location": "https://ms.immowelt.org/56094624-23c5-4ded-9aaa-a3b74850a63e/e273898e-9cf6-41d1-98b0-0595f86b50b6",
"MediaId": "56094624-23c5-4ded-9aaa-a3b74850a63e",
"MimeType": "image/jpeg",
"Position": 0,
"StoragePath": "/56094624-23c5-4ded-9aaa-a3b74850a63e/e273898e-9cf6-41d1-98b0-0595f86b50b6",
"Title": "Visualisierung"
},
{
"BaseUri": "https://ms.immowelt.org",
"FileName": "d3ee83b9-5ef5-45d1-ab31-12ac87dd0f6a__94189.jpg",
"FileSize": 2104607,
"Location": "https://ms.immowelt.org/78b40967-c4c7-4af9-ae05-66ced952a1b8/0e444f51-795c-4241-8564-2b1b1972078d",
"MediaId": "78b40967-c4c7-4af9-ae05-66ced952a1b8",
"MimeType": "image/jpeg",
"Position": 1,
"StoragePath": "/78b40967-c4c7-4af9-ae05-66ced952a1b8/0e444f51-795c-4241-8564-2b1b1972078d",
"Title": "Visualisierung"
},
{
"BaseUri": "https://ms.immowelt.org",
"FileName": "a2158f83-8ff1-4c4f-aabe-3346c5411733__94195.jpg",
"FileSize": 335528,
"Location": "https://ms.immowelt.org/673d7f44-6447-446a-a92c-61a919ac4654/b755630d-13be-4cc0-8741-94b97e096598",
"MediaId": "673d7f44-6447-446a-a92c-61a919ac4654",
"MimeType": "image/jpeg",
"Position": 2,
"StoragePath": "/673d7f44-6447-446a-a92c-61a919ac4654/b755630d-13be-4cc0-8741-94b97e096598",
"Title": "Lageplan"
},
{
"BaseUri": "https://ms.immowelt.org",
"FileName": "a9b788fe-6fc3-4b42-a474-986298864ba3__94197.jpg",
"FileSize": 369197,
"Location": "https://ms.immowelt.org/4d72d652-8af8-4e2c-82ef-00156f2ab6bd/b5b9d8e1-1123-47a3-9562-30dffccffe17",
"MediaId": "4d72d652-8af8-4e2c-82ef-00156f2ab6bd",
"MimeType": "image/jpeg",
"Position": 3,
"StoragePath": "/4d72d652-8af8-4e2c-82ef-00156f2ab6bd/b5b9d8e1-1123-47a3-9562-30dffccffe17",
"Title": "Lageplan"
},
{
"BaseUri": "https://ms.immowelt.org",
"FileName": "06cdce48-2973-4c2e-b51c-2e813a9350f4__94199.jpg",
"FileSize": 284088,
"Location": "https://ms.immowelt.org/6f6fc8d9-3ae5-4244-9ef5-6ed60137a7e0/7d7ea56d-56ad-4563-8e2a-ae7b5d522f04",
"MediaId": "6f6fc8d9-3ae5-4244-9ef5-6ed60137a7e0",
"MimeType": "image/jpeg",
"Position": 4,
"StoragePath": "/6f6fc8d9-3ae5-4244-9ef5-6ed60137a7e0/7d7ea56d-56ad-4563-8e2a-ae7b5d522f04",
"Title": "Freifl\u00e4chenplan"
},
{
"BaseUri": "https://ms.immowelt.org",
"FileName": "5b325dba-4e31-468d-ba68-25dbdfacc20f__94201.jpg",
"FileSize": 216346,
"Location": "https://ms.immowelt.org/0b38a06a-9783-4576-a0b4-8c82a1a33a06/c41efe0e-3024-4f17-8bd0-7e9af6635816",
"MediaId": "0b38a06a-9783-4576-a0b4-8c82a1a33a06",
"MimeType": "image/jpeg",
"Position": 5,
"StoragePath": "/0b38a06a-9783-4576-a0b4-8c82a1a33a06/c41efe0e-3024-4f17-8bd0-7e9af6635816",
"Title": "Grundriss TG"
},
{
"BaseUri": "https://ms.immowelt.org",
"FileName": "a82b73b6-7027-484c-8cad-2eef78ebe939__94203.jpg",
"FileSize": 92653,
"Location": "https://ms.immowelt.org/7dcc2481-8f60-45b0-8e28-5fef7479a7be/f3a7fb81-eedf-418e-b687-64cc633fcfbf",
"MediaId": "7dcc2481-8f60-45b0-8e28-5fef7479a7be",
"MimeType": "image/jpeg",
"Position": 6,
"StoragePath": "/7dcc2481-8f60-45b0-8e28-5fef7479a7be/f3a7fb81-eedf-418e-b687-64cc633fcfbf",
"Title": "Wohnung Nr. 6"
}
],
"Documents": [],
"OnlineId": "2CWC457",
"Visible": true,
"MediaId": 0,
"Offerer": {
"addressGuid": "B6CD62EE9FF644D19A24AE531A0DE223",
"globalUserId": 56453,
"sellerType": "partner",
"projectId": "iww",
"brokerValuation": {
"ReviewsEnabled": true,
"KeyStatistics": {
"TotalReviewCount": 0,
"AverageOverallRating": 0
}
},
"settings": {
"exposeHyperlinksAllowed": false,
"officeHours": ""
},
"contactData": {
"companyName": "I\u00b4M LIVING Immobilien",
"address": {
"city": "M\u00fcnchen",
"zipCode": "80539",
"street": "Maximilianstr. 40"
},
"profilePicture": "https://filestore.immowelt.de/ProfilBilder/150_526c9677a54740bbafcb531333130fc2.jpg",
"salutation": "Frau",
"firstName": "Christine",
"lastName": "Burmeister",
"position": "Selbst\u00e4ndige Immobilienmaklerin",
"mobile": "0160 / 187 86 08",
"phone": "089 / 121 90 680"
},
"immoweltExclusive": {
"exclusivityEnabled": false
},
"links": [
{
"Url": "https://www.immowelt.de/profil/56453",
"ExternalLink": false,
"LinkType": "BROKERPROFILE"
},
{
"Url": "https://www.immowelt.de/profil/56453",
"ExternalLink": false,
"LinkType": "COMPANYPROFILE"
},
{
"Url": "https://www.immowelt.de/profil/56453#objects",
"ExternalLink": false,
"LinkType": "OBJECTS"
},
{
"Url": "https://www.immowelt.de/profil/56453#reviews",
"ExternalLink": false,
"LinkType": "REVIEWS"
},
{
"Url": "https://www.immowelt.de/profil/56453#impressum",
"ExternalLink": false,
"LinkType": "IMPRESSUM"
},
{
"ExternalLink": true,
"Url": "http://www.i-m-living.de/",
"LinkType": "HOMEPAGE"
}
],
"logoUrl": "https://filestore.immowelt.de/logo/left_7dcf25d553bf43b4a6592bddc0be2beb.jpg"
},
"Project": {
"Address": {
"City": "M\u00fcnchen",
"CountryCode": "DE",
"PostCode": "80999",
"Street": "Theodor-Fischer-Strasse",
"HouseNumber": "99"
},
"AreaRange": {
"Max": 84,
"Min": 55
},
"ContactPersonGuid": 5159148,
"CreateDate": "2023-11-09T16:38:26.480Z",
"EstateObjectLinks": [
{
"GlobalObjectKey": "faacebad-82eb-4987-89ee-26feee19f01f"
},
{
"GlobalObjectKey": "6f1faa75-1228-4b40-ba6f-955ade7bb4e0"
},
{
"GlobalObjectKey": "a10f3a30-336f-44a6-88d6-2b2fba8627a1"
},
{
"GlobalObjectKey": "01c0c334-fd75-442a-bc86-83894df68d0a"
},
{
"GlobalObjectKey": "ef89aa6a-58a2-4dd9-9720-269e54a1bd11"
},
{
"GlobalObjectKey": "cb68ca03-f2a0-40ef-9479-ec7bcd3c839d"
}
],
"EstateObjectsCount": 8,
"Guid": 56453,
"Id": 6002,
"OnlineId": "k2kv732",
"PriceRange": {
"Currency": "EUR",
"Min": 599000,
"Max": 955000
},
"ProjectDescription": [
{
"Header": "Projektbeschreibung",
"Text": "Diese moderne und energetisch auf dem neusten Stand geplante Wohnung, wird in einem 8-Parteienhaus auf einem 799 m\u00b2 gro\u00dfen Grundst\u00fcck erstellt. Die gesamte Anlage wird bis Ende 2024 erbaut.\n\nDas Haus entsteht in massiver Bauweise und weist eine hochwertige Ausstattung auf. Die Wohnung wird schl\u00fcsselfertig, inklusiver der Au\u00dfenanlagen und Bepflanzung \u00fcbergeben.\n\nNehmen Sie alle Annehmlichkeiten eines Neubaus in Anspruch - wie Zahlung nach Baufortschritt - neuste Technik - Gew\u00e4hrleistung!\n\nIn diesem Haus stehen noch weitere Wohnungen zum Kauf zur Verf\u00fcgung.\n\nGerne unterst\u00fctzen wir Sie auch bei der Auswahl einer Finanzierung.\n\nDetails der Ausstattung:\n\n- PV Anlage zur Stromerzeugung f\u00fcr W\u00e4rmepumpe \n einschl. Batteriespeicher\n- Ziegelbauweise mit W\u00e4rmed\u00e4mmung\n- Elektrische Rolll\u00e4den mit Zeitschaltuhr\n- Fu\u00dfbodenheizung in allen Wohn- und Hobbyr\u00e4umen\n- Eiche-Parkettb\u00f6den\n- 3-fach Isolierverglasung\n- Beheizung mit moderner W\u00e4rmepumpe der Firma Viessmann \noder gleichwertig.\n- Bodengleiche Duschen mit Echtglas-Duschtrennw\u00e4nden\n- Baubegleitendes T\u00dcV Baucontrolling mit Dokumentation\n- Dezentrales Be- und Entl\u00fcftungssystem \n mit W\u00e4rmer\u00fcckgewinnung\n- Umfangreiche Elektroausstattung mit CAT-7 EDV Verkabelung\n\nWeitere Informationen stellen wir Ihnen gerne anhand der Baubeschreibung zur Verf\u00fcgung.\n\nGerne stehen wir f\u00fcr ein pers\u00f6nliches Gespr\u00e4ch oder einen Besichtigungstermin zur Verf\u00fcgung - wir freuen uns auf Ihre Kontaktaufnahme!"
}
],
"ProjectLogoLink": "https://ms.immowelt.org/9ce39a27-0e53-4ffb-b115-437a7ae681e0/50fcce08-3fdf-4e1b-9487-7f5a25be40d0",
"ProjectStatus": {
"ConstructionStartDate": "2023",
"ConstructionEndDate": "2024",
"MovingDate": "2024"
},
"RoomRange": {
"Max": 3,
"Min": 2
},
"Title": "Energieeffizente Wohnungen zum Eigennutz oder als Kapitalanlage"
}
}
]
Our immowelt.de scraper can successfully scrape property pages. Next, let's add
search page scraping to help us discover the properties we want to scrape.
Before we start scraping immowelt.de search pages, let's take a look at what the search pages look like. Search for any properties on the website and you will get a page similar to this:
We used the following URL to get the above search results:
https://www.immowelt.de/suche/muenchen/wohnungen/kaufen?sort=relevanz
To scrape search pages, we'll use the search API to get the data directly in JSON. Imoowelt has a hidden web API that is called when search requests are being made.
For a full introduction to hidden API scraping see this tutorial article which covers this subject in greater detail
To scrape this, we need to find the API responsible for fetching the search data. To do that, follow the steps below:
F12 key.Network tab and filter requests by clicking the Fetch/XHR button.By following the above steps, you should see all the requests sent from the browser to the search API:
We can see different XHR requests but we are only in the searches request. To see the response of this request, select it and click on the preview button:
We can also view the headers and body of the request by clicking on the Headers and Payload buttons:
The request body controls the search results, while the request headers represent the headers required by the server to authorize and validate the request.
To scrape this search API data, we'll replicate this request within our scraper by using the same request headers and payload. First, we'll start by setting up an httpx client with the headers and body same as the search API request:
# Add the authorization token below by getting a new one from developer tools
auth_token = ""
async def send_api_request(page_number: int):
"""get search data from the search API"""
# add the request headers
headers = {
"accept": "application/json, text/plain, */*",
"accept-language": "en-US,en;q=0.9",
# Add the authorization token below by getting a new one from developer tools as it may get expired
"authorization": f"Bearer {auth_token}",
"content-type": "application/json",
}
# add the request payload
data = {
"estateGroups": ["RESIDENTIAL", "COMMERCIAL"],
"estateType": "APARTMENT",
"distributionTypes": ["SALE"],
"estateSubtypes": [],
# locations ids of the search location, change them to search within a specific location
"locationIds": [4916],
# narrow down the search results by adding more search fields
"primaryPrice": {"min": None, "max": None},
"primaryArea": {"min": None, "max": None},
"areas": [{"areaType": "PLOT_AREA", "min": None, "max": None}],
"rooms": {"min": None, "max": None},
"constructionYear": {"min": None, "max": None},
"geoRadius": {"radius": None, "point": {"lat": None, "lon": None}},
"sort": {"direction": "DESC", "field": "RELEVANCE"},
"immoItemTypes": ["ESTATE", "PROJECT"],
"paging": {"size": 20, "page": page_number},
}
# create a new httpx client with new headers and payload values
client = AsyncClient(headers=headers)
# send a POST request to the search API
response = await client.post(
url="https://api.immowelt.com/residentialsearch/v1/searches",
data=json.dumps(data)
)
return response
# Add the authorization token below by getting a new one from developer tools
auth_token = ""
async def send_api_request(page_number: int):
"""get search data from the search API"""
# add the request headers
headers = {
"accept": "application/json, text/plain, */*",
"accept-language": "en-US,en;q=0.9",
# Add the authorization token below by getting a new one from developer tools as it may get expired
"authorization": f"Bearer {auth_token}",
"content-type": "application/json",
}
# add the request payload
data = {
"estateGroups": ["RESIDENTIAL", "COMMERCIAL"],
"estateType": "APARTMENT",
"distributionTypes": ["SALE"],
"estateSubtypes": [],
# locations ids of the search location, change them to search within a specific location
"locationIds": [4916],
# narrow down the search results by adding more search fields
"primaryPrice": {"min": None, "max": None},
"primaryArea": {"min": None, "max": None},
"areas": [{"areaType": "PLOT_AREA", "min": None, "max": None}],
"rooms": {"min": None, "max": None},
"constructionYear": {"min": None, "max": None},
"geoRadius": {"radius": None, "point": {"lat": None, "lon": None}},
"sort": {"direction": "DESC", "field": "RELEVANCE"},
"immoItemTypes": ["ESTATE", "PROJECT"],
"paging": {"size": 20, "page": page_number},
}
# send a POST request to the search API
response = await scrapfly.async_scrape(ScrapeConfig(
url="https://api.immowelt.com/residentialsearch/v1/searches",
headers=headers,
body=json.dumps(data),
# set the proxy country to Germany
country="DE",
method="POST"
))
return response
Here, we define a send_api_request function to send a POST requests to the immowelt.de search API. Next, we'll use this function to crawl over search pages and parse the JSON response:
import asyncio
import json
from typing import List, Dict
from httpx import AsyncClient, Response
# Add the authorization token below by getting a new one from developer tools as it may get expired
auth_token = ""
async def send_api_request(page_number: int):
# the remaining of the function logic here
def parse_search_api(response: Response) -> List[Dict]:
"""parse the search API response"""
data = json.loads(response.text)
search_data = data["data"]
max_search_pages = data["pagesCount"]
return {
"search_data": search_data,
"max_search_pages": max_search_pages
}
async def scrape_search(scrape_all_pages: bool, max_scrape_pages: int) -> List[Dict]:
"""scrape search pages usign the search API"""
# scrape the first search page
first_page = await send_api_request(1)
data = parse_search_api(first_page)
search_data = data["search_data"]
# get the the number of available search pages
max_search_pages = data["max_search_pages"]
# scrape all available pages in the search if scrape_all_pages = True or max_pages > total_search_pages
if scrape_all_pages == False and max_scrape_pages < max_search_pages:
max_scrape_pages = max_scrape_pages
else:
max_scrape_pages = max_search_pages
# scrape the remaining search pages from the API
for page in range(2, max_scrape_pages + 1):
response = await send_api_request(page)
data = parse_search_api(response)["search_data"]
search_data.extend(data)
print(f"scraped {len(search_data)} properties from search")
return search_data
import json
import asyncio
from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse
from typing import Dict, List
scrapfly = ScrapflyClient(key="Your ScrapFly API key")
# Add the authorization token below by getting a new one from developer tools
auth_token = ""
async def send_api_request(page_number: int):
# the remaining of the function logic here
def parse_search_api(response: ScrapeApiResponse) -> List[Dict]:
"""parse the search API response"""
data = json.loads(response.scrape_result["content"])
search_data = data["data"]
max_search_pages = data["pagesCount"]
return {
"search_data": search_data,
"max_search_pages": max_search_pages
}
async def scrape_search(scrape_all_pages: bool, max_scrape_pages: int) -> List[Dict]:
"""scrape search pages usign the search API"""
# scrape the first search page
first_page = await send_api_request(1)
data = parse_search_api(first_page)
search_data = data["search_data"]
# get the the number of available search pages
max_search_pages = data["max_search_pages"]
# scrape all available pages in the search if scrape_all_pages = True or max_pages > total_search_pages
if scrape_all_pages == False and max_scrape_pages < max_search_pages:
max_scrape_pages = max_scrape_pages
else:
max_scrape_pages = max_search_pages
# scrape the remaining search pages from the API
for page_number in range(2, max_scrape_pages + 1):
response = await send_api_request(page_number)
data = parse_search_api(response)["search_data"]
search_data.extend(data)
print(f"scraped {len(search_data)} properties from search")
return search_data
if __name__ == "__main__":
search_data = asyncio.run(scrape_search(
scrape_all_pages=False,
max_scrape_pages=3
))
# print the result in JSON format
print(json.dumps(search_data, indent=2))
The result is a list containing data from three search API pages:
[
{
"id": "project_4401",
"itemType": "PROJECT",
"projectId": "4401",
"locationIds": [
529709,
529492,
528418,
464967,
4916,
115926,
122156,
121726
],
"place": {
"city": "München"
},
"estateTypes": [
"APARTMENT"
],
"estateSubtypes": [],
"distributionType": "SALE",
"title": " An der Rehwiese 3 & 9 - München",
"prices": [
{
"amountMin": 709600.0,
"amountMax": 1024500.0,
"type": "PURCHASE_PRICE",
"currency": "EUR",
"attributes": []
}
],
"areas": [
{
"sizeMin": 55.0,
"sizeMax": 114.0,
"type": "LIVING_AREA"
}
],
"primaryArea": {
"sizeMin": 55.0,
"sizeMax": 114.0,
"type": "LIVING_AREA"
},
"primaryPrice": {
"amountMin": 709600.0,
"amountMax": 1024500.0,
"type": "PURCHASE_PRICE",
"currency": "EUR",
"attributes": []
},
"roomsMin": 2.0,
"roomsMax": 4.0,
"timestamp": "2022-10-31T12:23:08.502Z",
"isNew": false,
"features": [
"NEW_BUILDING",
"NEW_BUILDING_PROJECT"
],
"pictures": [
{
"imageUri": "https://ms.immowelt.org/4c985c53-c27d-4dc0-9eaf-12b9cf7ffef8/7fcce537-d7ee-4212-8602-c95e3a24d531",
"imageUriBasePath": "https://ms.immowelt.org/4c985c53-c27d-4dc0-9eaf-12b9cf7ffef8/7fcce537-d7ee-4212-8602-c95e3a24d531"
},
{
"imageUri": "https://ms.immowelt.org/9b9243cb-d7c7-45c4-bfd7-3afdcf43a555/74cda368-a126-4321-b868-180fba731af9",
"imageUriBasePath": "https://ms.immowelt.org/9b9243cb-d7c7-45c4-bfd7-3afdcf43a555/74cda368-a126-4321-b868-180fba731af9"
}
],
"presentationProducts": [],
"broker": {
"guid": 28000,
"sellerType": "PARTNER",
"companyName": "Stauch Wohnbau GmbH",
"partnerAward": "PLATIN",
"immoweltExclusive": false,
"logoUri": "//filestore.immowelt.de/logo/100_afa77f7be8254db1b48ff73941b6d194.jpg",
"logoUriHttps": "https://filestore.immowelt.de/logo/100_afa77f7be8254db1b48ff73941b6d194.jpg"
},
"onlineId": "k2jb632",
"projectData": {
"constructionStartDate": "2022",
"constructionEndDate": "2024",
"movingDate": "2024",
"logoUri": "https://ms.immowelt.org/516ff6e5-b724-4f3f-8232-07c7280172db/95a8a292-d10b-4048-b619-3991ccc0e769",
"numberOfUnits": 18
},
"relatedItems": [
{
"itemType": "ESTATE",
"onlineId": "2ble55s",
"gok": "f7c6c447-aacd-46b0-bb44-9cb8b5b5fd97",
"projectId": null,
"title": null,
"roomsMin": 3.5,
"roomsMax": 3.5,
"primaryPrice": {
"amountMin": 939200.0,
"amountMax": 939200.0,
"type": "PURCHASE_PRICE",
"currency": "EUR",
"attributes": []
},
"primaryArea": {
"sizeMin": 99.0,
"sizeMax": 99.0,
"type": "LIVING_AREA"
},
"features": [
"GARDEN_SHARED",
"ATTIC",
"CELLAR_SHARE",
"BATH_WITH_TUB",
"BATH_WITH_WINDOW",
"PASSENGER_LIFT",
"TERRACE",
"UNDERGROUND_PARKING",
"PARKING_AREA",
"FREE_OF_COMMISSION"
]
}
],
"estateGroups": [
"RESIDENTIAL"
]
}
]
Our immowelt.de scraper can successfully scrape property and search pages. However, we may start getting blocked after sending additional requests. Let's look at a solution!
To avoid scraping blocking on immowelt.de, we'll use ScrapFly. A web scraping API that scales up web scraper by providing:
Here is how to use the asp feature with the ScrapFly Python SDK to scrape immowelt.de without getting blocked:
import httpx
from parsel import Selector
response = httpx.get("some immowelt.de url")
selector = Selector(response.text)
# in ScrapFly SDK becomes
from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse
scrapfly_client = ScrapflyClient("Your ScrapFly API key")
result: ScrapeApiResponse = scrapfly_client.scrape(ScrapeConfig(
# some immowelt.de URL
url="https://www.immowelt.de/expose/2czpn55",
# select a specific proxy country
country="DE",
# enable anti scraping protection bypass
asp=True,
# allows JavaScript rendering similar to headless browsers
render_js=True
))
# use the built-in parsel selector
selector = result.selector
To wrap up this guide, let's take a look at some frequently asked questions about web scraping immowelt.de.
Yes, all the data on immowelt.de is publicly available and it's legal to scrape them as long as you follow the website's terms of service and keep your scraping rate reasonable. For further details, refer to our previous guide - is web scraping legal.
At the time of writing, there is no public API for immowelt.de. However, we have seen that's possible to use the private immowelt.de search API to scrape real estate data.
Yes, immobilienscout24.de is another popular website for real estate ads in Germany. We have covered web scraping immobilienscout24.de in a previous article. Refer to the #realestate scraping tag for more real estate property scraping targets.
Another popular target for real estate property scraping in Germany is Immobilienscout24.de and we have a guide for it as well!
In this article, we explained how to scrape immowelt.de with Python - a popular website for real estate ads in Germany.
We went through a step-by-step guide on scraping immowelt.de real estate data from property pages and search API. Furthermore, we explained how to avoid immowelt.de web scraping blocking.
In this article, we'll explore how to scrape Reddit. We'll extract various social data types from subreddits, posts, and user pages. All of which through plain HTTP requests without headless browser usage.
In this scrape guide we'll be taking a look at one of the most popular web scraping targets - LinkedIn.com. We'll be scraping people profiles, company profiles as well as job listings and search.
In this guide, we'll explain how to scrape SimilarWeb through a step-by-step guide. We'll scrape comprehensive website traffic insights, websites comparing data, sitemaps, and trending industry domains.