How to Scrape Immowelt.de Real Estate Data

article feature image

Immowelt.de is a popular website for real estate ads in Germany, featuring thousands of real estate property ads.

In this article, we'll cover how to scrape real estate data from immowelt.de property pages and search API. We'll also explain how to avoid immowelt.de web scraping blocking. Let's dive in!

Latest immowelt.de Scraper Code

https://github.com/scrapfly/scrapfly-scrapers/

Why Scrape Immowelt.de?

Exploring thousands of property listings on Immowelt.de manually can be time-consuming. However, web scraping can extract loads of data with just a few code lines, saving a lot of time and resources.

Scrapping immowelt.de enables investors and buyers to explore potential investment opportunities by discovering low-priced properties with high growth potential.

Immowelt.de web scraping can help real estate businesses monitor competitors' listings and pricing strategies, allowing for remaining up-to-date with the market changes and staying competitive.

Project Setup

In this guide on web scraping immowelt.de, we'll use a few Python libraries:

  • httpx: For sending HTTP requests to the website.
  • parsel: For parsing the HTML using XPath and CSS selectors.
  • asyncio: For runnA Python SDK for ScrapFly, a web scraping API that allows for scraping at scale and bypassing any web scraping blocking.
  • scrapfly-sdk: A Python SDK for ScrapFly, a web scraping API that allows for scraping at scale and bypassing any web scraping blocking.

You don't need to install asyncio, as it comes pre-installed in Python. Install the other libraries using the following pip command:

pip install httpx parsel scrapfly-sdk

How to Scrape Immowelt.de Property Pages?

Let's begin by scraping immowelt.de property pages. Go to any property page on the website and you will get a page similar to this:

property page on immowelt.de
Property page on Immowelt.de

Instead of scraping this page content using selectors like CSS and XPath, we'll extract the data directly in JSON from script tags. To see this data, open browser developer tools and scroll down to the script tag with the id serverApp-state:

hidden web data unter script tags
Hidden web data unter script tags

The data inside this tag is the same data seen on the web page but before getting rendered into the HTML. Which is often known as hidden web data. However, this data isn't a valid JSON yet as it's encoded with &q; characters. To scrape it, we'll select this script tag and decode the data inside:

Python
ScrapFly
import asyncio
import json
from typing import Dict, List
from httpx import AsyncClient, Response
from parsel import Selector

client = AsyncClient(
    headers={
        # use same headers as a popular web browser (Chrome on Windows in this case)
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
        "Accept-Language": "en-US,en;q=0.9",
    }
)

def parse_next_data(response: Response) -> Dict:
    """parse lsiting data from property page"""
    # get the property id from the page URL
    proeprty_id = str(response.url).split(".de/")[-1]
    selector = Selector(response.text)
    # extract the data from the script tag
    next_data = selector.xpath("//script[@id='serverApp-state']/text()").get()
    # parse the data into a valid JSON object
    next_data = json.loads(next_data.replace("&q;", '"').replace("\n", " "))
    return next_data[proeprty_id]

async def scrape_properties(urls: List[str]) -> List[Dict]:
    """scrape listings data from property pages"""
    # add all property pages to a scraping list
    to_scrape = [client.get(url) for url in urls]
    properties = []
    # scrape all property pages concurrently
    for response in asyncio.as_completed(to_scrape):
        try:
            data = parse_next_data(await response)
            properties.append(data)
        except:
            print("expired property page")
    print(f"scraped {len(properties)} property pages")
    return properties    
import asyncio
import json
from typing import Dict, List
from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse

scrapfly = ScrapflyClient(key="Your ScrapFly API key")

def parse_next_data(response: ScrapeApiResponse) -> Dict:
    """parse lsiting data from property page"""
    # get the property id from the page URL
    proeprty_id = str(response.context["url"]).split(".de/")[-1]
    selector = response.selector
    # extract the data from the script tag
    next_data = selector.xpath("//script[@id='serverApp-state']/text()").get()
    # parse the data into a valid JSON object
    next_data = json.loads(next_data.replace("&q;", '"').replace("\n", " "))
    return next_data[proeprty_id]

async def scrape_properties(urls: List[str]) -> List[Dict]:
    """scrape listings data from property pages"""
    # add all property pages to a scraping list
    to_scrape = [ScrapeConfig(url, asp=True, country="DE") for url in urls]
    properties = []
    # scrape all property pages concurrently
    async for response in scrapfly.concurrent_scrape(to_scrape):
        try:
            data = parse_next_data(response)
            properties.append(data)
        except:
            print("expired property page")
    print(f"scraped {len(properties)} property pages")
    return properties    
Run the code
if __name__ == "__main__":
    data = asyncio.run(scrape_properties(
        urls = [
            "https://www.immowelt.de/expose/27t9c5f",
            "https://www.immowelt.de/expose/27dgc5f",
            "https://www.immowelt.de/expose/25jqw5t",
            "https://www.immowelt.de/expose/249p65w",
            "https://www.immowelt.de/expose/24c7p5k"
        ]
    ))
    # print the data in JSON format
    print(json.dumps(data, indent=2))    

We start by creating an async httpx client with standard web browser headers. We then define our scraping functions:

  • parse_next_data() function to select and parse the hidden web data into a valid JSON object.
  • scrape_properties() function to scrape the property pages by adding their URLs to a scraping list and concurrently scraping them.

The result is a list containing all the property page data:

Run the code
[
    {
        "GlobalObjectKey": "01c0c334-fd75-442a-bc86-83894df68d0a",
        "CreateDate": "2023-10-23T12:09:08.723Z",
        "General": {
            "Headline": "Moderne, hochwertige Eigentumswohnung  mit Grundwasserw\u00e4rmepumpe und Photovoltaikanlage",
            "EstateType": "APARTMENT",
            "EstateTypeKey": "WOHNUNG",
            "ReferenceNumber": "CB.2309",
            "DistributionTypeKey": "ZUM_KAUF",
            "DistributionType": "BUY",
            "Rooms": 2,
            "LivingSpace": 54.7
        },
        "Seo": {
            "BreadCrumb": [
                {
                    "Id": "https://www.immowelt.de/",
                    "Name": "Immobilien"
                },
                {
                    "Id": "https://www.immowelt.de/suche/wohnungen",
                    "Name": "Wohnungssuche"
                },
                {
                    "Id": "https://www.immowelt.de/suche/wohnungen/kaufen",
                    "Name": "Kaufen"
                },
                {
                    "Id": "https://www.immowelt.de/suche/muenchen-allach-untermenzing/wohnungen/kaufen",
                    "Name": "M\u00fcnchen (Allach-Untermenzing)"
                },
                {
                    "Id": "https://www.immowelt.de/liste/muenchen-allach-untermenzing/wohnungen/kaufen?sort=relevanz",
                    "Name": "Ergebnisliste"
                }
            ],
            "MetaTags": {
                "PageTitle": "Moderne, hochwertige Eigentumswohnung  mit Grundwasserw\u00e4rmepumpe und Photovoltaikanlage | Wohnung M\u00fcnchen",
                "Description": "Wohnung in M\u00fcnchen zum Kauf mit 2 Zimmer und 54,7 m\u00b2 Wohnfl\u00e4che. Ausstattung: barrierefrei, Kelleranteil, Personenaufzug, Erstbezug, Fu\u00dfbodenheizung, Zentralheizung."
            },
            "SeoOptimizedLocation": "muenchen-allach-untermenzing"
        },
        "Tealium": {
            "enh_promo_id": [
                "kfw",
                "interhyp",
                "treppenmeister"
            ],
            "enh_promo_name": [
                "kfw",
                "interhyp_text",
                "treppenmeister"
            ],
            "enh_promo_creative": [
                "expose_teaser",
                "expose_servicebereich",
                "expose_teaser"
            ],
            "enh_promo_position": [
                "expose",
                "expose",
                "expose"
            ],
            "broker_guid": "56453",
            "object_count_photos": 7,
            "object_state": [
                "Deutschland"
            ],
            "object_locationid": "464951",
            "object_city": [
                "M\u00fcnchen"
            ],
            "object_district": [
                "Allach-Untermenzing"
            ],
            "object_federalstate": [
                "Bayern"
            ],
            "object_county": [
                "M\u00fcnchen"
            ],
            "object_zip": "80999",
            "object_address_is_visible": false,
            "object_price": 599000,
            "object_currency": "EUR",
            "object_gok": [
                "01c0c334-fd75-442a-bc86-83894df68d0a"
            ],
            "object_features": [
                "barriefrei",
                "Kelleranteil",
                "Personenaufzug",
                "ERSTBEZUG",
                "Fu\u00dfbodenheizung",
                "Zentralheizung",
                "Tiefgarage",
                "wasch_trockenraum",
                "kable_sat_tv"
            ],
            "object_rooms": 2,
            "object_area": 54.7,
            "object_marketingtype": [
                "KAUF"
            ],
            "object_objektart": [
                "wohnung"
            ],
            "object_objektart_sub": "wohnung",
            "product_id": [
                "500"
            ],
            "product_name": [
                "kontakt"
            ],
            "product_price": [
                0
            ],
            "product_price_total": 0,
            "product_variant": [
                ""
            ],
            "product_quantity": [
                1
            ],
            "product_period": [
                "0"
            ],
            "product_payment": [
                ""
            ],
            "product_coupon": "",
            "product_discount": [
                0
            ],
            "product_net_price": [
                0
            ],
            "product_net_price_total": "0.00",
            "product_business_type": "living",
            "product_building_state": "FIRST_TIME_USE",
            "expose_type": "project",
            "page_type": "productdetailpage",
            "app_medienid": "immowelt.de"
        },
        "InternalAds": [
            {
                "Id": "TREPPENMEISTER",
                "URL": "https://www.immowelt.de/customer/treppenmeister/"
            }
        ],
        "Advertisement": {
            "TargetValues": {
                "IMMO_ADGROUP": "IMMOBILIEN_EXPOSE",
                "IMMO_ART": 1,
                "IMMO_ESTATETYPE": "APARTMENT",
                "IMMO_MK": 1,
                "IMMO_DISTRIBUTION": "BUY",
                "IMMO_NT": 13,
                "IMMO_P": "599000",
                "IMMO_PRANGE": "500000-1000000",
                "IMMO_GLOBALUSERID": 56453,
                "IMMO_LOCATIONID": 464951,
                "IMMO_ORTCLEAN": "M\u00fcnchen",
                "IMMO_ORT": "M\u00fcnchen (Allach-Untermenzing)",
                "IMMO_PLZ": "80999",
                "IMMO_TITLE": "Moderne, hochwertige Eigentumswohnung  mit Grundwasserw\u00e4rmepumpe und Photovoltaikanlage",
                "IMMO_BILD": "https://ms.immowelt.org/56094624-23c5-4ded-9aaa-a3b74850a63e/e273898e-9cf6-41d1-98b0-0595f86b50b6",
                "IMMO_ZA": 2,
                "IMMO_WFL": 54.7
            },
            "CampaignLinks": [
                "UMZUG",
                "ODA",
                "KFW"
            ],
            "IsAllowedForFinancing": true
        },
        "EquipmentAreas": [
            {
                "Key": "APARTMENT",
                "Headline": "Die Wohnung",
                "Equipments": [
                    {
                        "Key": "FLOOR",
                        "Value": "3. Geschoss",
                        "DisplayType": "TABLE",
                        "Label": "Wohnungslage"
                    },
                    {
                        "Key": "SUITABILITY",
                        "Value": "barrierefrei",
                        "DisplayType": "LIST",
                        "Label": ""
                    },
                    {
                        "Key": "CONDITION",
                        "Value": "Erstbezug",
                        "DisplayType": "LIST",
                        "Label": "Zustand"
                    },
                    {
                        "Key": "OTHERROOMS",
                        "Value": "Kelleranteil, Wasch-Trockenraum",
                        "DisplayType": "LIST",
                        "Label": "Weitere R\u00e4ume"
                    },
                    {
                        "Key": "CONNECTIVITY",
                        "Value": "Kabelanschluss",
                        "DisplayType": "LIST",
                        "Label": "Anschl\u00fcsse"
                    }
                ]
            },
            {
                "Key": "SECONDARY",
                "Headline": "Wohnanlage",
                "Equipments": [
                    {
                        "Key": "ELEVATOR",
                        "Value": "Personenaufzug",
                        "DisplayType": "LIST",
                        "Label": ""
                    },
                    {
                        "Key": "PARKINGSLOT",
                        "Value": "Tiefgarage",
                        "DisplayType": "LIST",
                        "Label": "1 Stellplatz"
                    }
                ]
            },
            {
                "Key": "ENERGY",
                "Headline": "Energie / Versorgung",
                "Equipments": [
                    {
                        "Key": "HEATING",
                        "Value": "Fu\u00dfbodenheizung, Zentralheizung",
                        "DisplayType": "LIST",
                        "Label": "Heizungsart"
                    }
                ]
            }
        ],
        "HardFacts": [
            {
                "NumberValue": 599000,
                "Unit": "EUR",
                "Key": "PRICE",
                "Label": "Kaufpreis"
            },
            {
                "Key": "AREA_LIVING",
                "Label": "Wohnfl\u00e4che",
                "NumberValue": 54.7,
                "Unit": "SQM",
                "Comments": [
                    "ca."
                ]
            },
            {
                "Key": "ROOMS",
                "Label": "Zimmer",
                "NumberValue": 2
            }
        ],
        "MetaBadges": [],
        "LocalRatings": {
            "lastUpdated": "2023-11-06T17:05:22.000Z",
            "scores": {
                "local_amenities": 1.6,
                "mobility": 7.4
            }
        },
        "Texts": [
            {
                "Title": "Lage",
                "Content": "Untermenzing ist ein Stadtteil im M\u00fcnchner Westen und geh\u00f6rt ebenfalls zum Stadtbezirk Pasing-Obermenzing. Der Stadtteil liegt direkt am Ufer der W\u00fcrm und ist von viel Gr\u00fcn umgeben.\n\nUntermenzing ist vor allem durch seine ruhige und l\u00e4ndliche Lage bekannt. Der Stadtteil besteht haupts\u00e4chlich aus Einfamilienh\u00e4usern und kleinen Wohnanlagen. Die Stra\u00dfen von B\u00e4umen ges\u00e4umt, was dem Viertel eine idyllische Atmosph\u00e4re verleiht.\n\nDas Zentrum von Untermenzing bildet der historische Ortskern mit der Pfarrkirche St. Michael und dem alten Rathaus. Hier gibt es einige Gesch\u00e4fte, Restaurants und Caf\u00e9s, die zum Verweilen einladen. Auch der Wochenmarkt, der regelm\u00e4\u00dfig stattfindet, ist ein beliebter Treffpunkt f\u00fcr die Bewohner.\n\nNeben dem Ortskern bietet Untermenzing auch zahlreiche Gr\u00fcnfl\u00e4chen und Naherholungsgebiete. Der nahegelegene W\u00fcrmkanal und der Aubinger Lohe sind beliebte Ziele f\u00fcr Spazierg\u00e4nge und Radtouren. Auch der nahegelegene Langwieder See bietet M\u00f6glichkeiten f\u00fcr Freizeitaktivit\u00e4ten im Gr\u00fcnen.\n\nDie Verkehrsanbindung in Untermenzing ist gut. Mit der S-Bahn-Linie S2 ist man in etwa 20 Minuten am M\u00fcnchner Hauptbahnhof. Auch die Autobahn A8 ist schnell zu erreichen, sodass man schnell in Richtung Augsburg oder Stuttgart fahren kann.\n\nInsgesamt ist Untermenzing ein ruhiger und naturnaher Stadtteil, der vor allem durch seine l\u00e4ndliche Atmosph\u00e4re und die gute Anbindung an die Innenstadt \u00fcberzeugt.",
                "Position": 1
            },
            {
                "Title": "Objektbeschreibung",
                "Content": "Die Neubauwohnung ist eine moderne und stilvolle Wohnung, die derzeit fertiggestellt wird. Sie befindet sich in einem Mehrfamilienhaus, das auf einem ca. 799 m\u00b2 gro\u00dfen Grundst\u00fcck steht. Sie bietet alle Annehmlichkeiten eines zeitgem\u00e4\u00dfen Wohnens.\n\nDie Wohnung verf\u00fcgt \u00fcber gro\u00dfz\u00fcgige und lichtdurchflutete R\u00e4ume, die mit hochwertigen Materialien, wie z. B. Parkettboden und Fu\u00dfbodenheizung ausgestattet ist. Der offene Grundriss schafft eine angenehme Wohnatmosph\u00e4re und erm\u00f6glicht eine flexible Gestaltung der Wohnbereiche.\n\nDas Badezimmer ist modern und elegant gestaltet und verf\u00fcgt \u00fcber eine ebenerdige Dusche mit Echtglas-Duschtrennwand.\n\nDie Wohnung befindet sich im Obergeschoss und verf\u00fcgt \u00fcber einen Balkon, auf dem man die frische Luft genie\u00dfen und den Ausblick auf die Umgebung genie\u00dfen kann. \n\nZur Ausstattung der Wohnung geh\u00f6ren auch ein Aufzug, ein Kellerabteil und ein Parkplatz in der Tiefgarage. Die Wohnung ist gut isoliert und verf\u00fcgt \u00fcber eine effiziente Heizungsanlage, um den Energieverbrauch zu minimieren.\n\nDie Neubauwohnung befindet sich in einer attraktiven Lage mit guter Anbindung an \u00f6ffentliche Verkehrsmittel und Einkaufsm\u00f6glichkeiten. Sie bietet eine moderne und komfortable Wohnl\u00f6sung f\u00fcr Singles, Paare oder Familien, die ein zeitgem\u00e4\u00dfes und stilvolles Zuhause suchen.\n\nDas Haus entsteht in massiver Bauweise und die Wohnung wird schl\u00fcsselfertig, inklusiver der Au\u00dfenanlagen und Bepflanzung \u00fcbergeben.\n\nNehmen Sie alle Annehmlichkeiten eines Neubaus in Anspruch - wie Zahlung nach Baufortschritt - neuste Technik - Gew\u00e4hrleistung!\n\nDie Einhaltung des hohen Qualit\u00e4tsanspruches, der an die einzelnen Handwerksbetriebe gestellt wird, wird vom T\u00dcV S\u00dcD in Form eines Baucontrollings \u00fcberwacht. Der Bautr\u00e4ger, die Forma Hausbau GmbH hat in der Umgebung von M\u00fcnchen bereits zahlreiche Bauprojekte verwirklicht.\n\nIn diesem Haus stehen noch weitere Wohnungen zum Kauf zur Verf\u00fcgung.\n\nGerne unterst\u00fctzen wir Sie auch bei der Auswahl einer Finanzierung.",
                "Position": 2
            },
            {
                "Title": "Ausstattung",
                "Content": "Details der Ausstattung:\n\n- PV Anlage zur Stromerzeugung f\u00fcr W\u00e4rmepumpe \n einschl. Batteriespeicher\n- Ziegelbauweise mit W\u00e4rmed\u00e4mmung\n- Elektrische Rolll\u00e4den mit Zeitschaltuhr\n- Fu\u00dfbodenheizung in allen Wohn- und Hobbyr\u00e4umen\n- Eiche-Parkettb\u00f6den\n- 3-fach Isolierverglasung\n- Beheizung mit moderner W\u00e4rmepumpe der Firma Viessmann \noder gleichwertig.\n- Bodengleiche Duschen mit Echtglas-Duschtrennw\u00e4nden\n- Baubegleitendes T\u00dcV Baucontrolling mit Dokumentation\n- Dezentrales Be- und Entl\u00fcftungssystem \n mit W\u00e4rmer\u00fcckgewinnung\n- Umfangreiche Elektroausstattung mit CAT-7 EDV Verkabelung\n\nWeitere Informationen stellen wir Ihnen gerne anhand der Baubeschreibung zur Verf\u00fcgung.\n\nGerne stehen wir f\u00fcr ein pers\u00f6nliches Gespr\u00e4ch oder einen Besichtigungstermin zur Verf\u00fcgung - wir freuen uns auf Ihre Kontaktaufnahme!",
                "Position": 3
            },
            {
                "Title": "Preisinformation",
                "Content": "1 Tiefgaragenstellplatz, Kaufpreis: 30.000,00 EUR\n",
                "Position": 4
            },
            {
                "Title": "Weitere Informationen",
                "Content": "Sonstiges\nEs liegt ein Energiebedarfsausweis vor.\nDieser ist g\u00fcltig bis 9.5.2023.\nEndenergiebedarf betr\u00e4gt 10.40 kwh/(m\u00b2*a).\nWesentlicher Energietr\u00e4ger der Heizung ist Luft/wasser W\u00e4rmepumpe.\nDie Energieeffizienzklasse ist A+.\n\nSonstige Angaben:\nIhre Ansprechpartnerin:\nChristine Burmeister\n\nTel. 0160-1878608\n\nErlaubnis nach \u00a734c der Gewerbeordnung erteilt durch das Landratsamt Starnberg, Strandbadstr. 2, 82319 Starnberg.\n\nAlle Angaben sind ohne Gew\u00e4hr und basieren ausschlie\u00dflich auf Informationen, die uns vom Auftraggeber zur Verf\u00fcgung gestellt wurden. Wir \u00fcbernehmen keine Gew\u00e4hr f\u00fcr die Vollst\u00e4ndigkeit, Richtigkeit und Aktualit\u00e4t dieser Angaben. Die Nutzung bzw. Weitergabe s\u00e4mtlicher Bilder und grafischen Darstellungen ist nicht gestattet. Mit der Anfrage auf dieses Objekt erkl\u00e4rt der Interessent seine ausdr\u00fcckliche Einwilligung f\u00fcr die Kontaktaufnahme per Telefon oder e-Mail durch einen unserer Mitarbeiter.\n\nStichworte\nTiefgarage vorhanden, Nutzfl\u00e4che: 6,60 m\u00b2, vermietbare Fl\u00e4che: 54,70 m\u00b2, Anzahl der Schlafzimmer: 1, Anzahl der Badezimmer: 1, Anzahl Balkone und Terrassen: 1, Balkon-Terrassen-Fl\u00e4che: 6,40 m\u00b2, Kellerfl\u00e4che: 3,40 m\u00b2",
                "Position": 5
            }
        ],
        "Price": {
            "AdditionalInformation": {
                "Commission": {
                    "CommissionType": "COMMISSION_DEFAULT",
                    "DisplayValue": {
                        "Label": "Provision f\u00fcr K\u00e4ufer",
                        "StringValue": "Bitte beachte, das Angebot kann bei Vertragsabschluss die Zahlung einer Provision beinhalten. Weitere Informationen erh\u00e4ltst Du vom Anbieter."
                    }
                },
                "MarketPricing": {
                    "Heading": "Wohnungspreise in M\u00fcnchen (Allach-Untermenzing)",
                    "Link": "https://www.immowelt.de/immobilienpreise/muenchen-allach-untermenzing/wohnungspreise"
                }
            },
            "DataTable": [
                {
                    "NumberValue": 599000,
                    "Unit": "EUR",
                    "Key": "PRICE_BUYING_PRICE",
                    "Label": "Kaufpreis"
                },
                {
                    "NumberValue": 30000,
                    "Unit": "EUR",
                    "Key": "PRICE_PARKINGPRICE",
                    "Label": "1 Stellplatz"
                }
            ]
        },
        "EstateAddress": {
            "Country": "Deutschland",
            "City": "M\u00fcnchen",
            "District": "Allach-Untermenzing",
            "ZipCode": "80999",
            "LocationId": 464951,
            "PublishStreet": false,
            "FederalState": "Bayern",
            "FederalStateId": 2
        },
        "EstateMapData": {
            "LocationCoordinates": {
                "Latitude": 48.19266,
                "Longitude": 11.46333
            },
            "ShowPin": false,
            "LocationId": 464951,
            "LocatedIn": "DE",
            "Locations": [
                {
                    "latitude": 48.19266,
                    "longitude": 11.46333,
                    "locationId": 464951,
                    "name": "Allach-Untermenzing",
                    "adminTypeName": "Stadtteil",
                    "locationTypeId": 10
                },
                {
                    "locationId": 4916,
                    "name": "M\u00fcnchen",
                    "adminTypeName": "Kreisfreie Stadt",
                    "locationTypeId": 6
                },
                {
                    "locationId": 115926,
                    "name": "Oberbayern",
                    "adminTypeName": "Regierungsbezirk",
                    "locationTypeId": 5
                },
                {
                    "locationId": 122156,
                    "name": "Bayern",
                    "adminTypeName": "Bundesland",
                    "locationTypeId": 4
                },
                {
                    "locationId": 121726,
                    "name": "Deutschland",
                    "adminTypeName": "Land",
                    "locationTypeId": 2
                }
            ],
            "Polygons": [
                {
                    "polygon": "q}eeHsmaeAyKoKiS`DiEyHuI{@yAbAs@xBmBpJwAbQ}HiBaGqE{M~_@RdKrBzKhXlu@~CdEbEhD~C@hAl@dArEdJvC|CnBvIfhAv@pHbBxFj@dWrRvFxDMlThApNoAzH}DrWyDtKlMd@~]nEnHlKbFrHwCnHyLrLGdCmNjDgLvCeEzFeFvFcHbA{BdC}GaAgPxCsFzBi@wA}T{BuDiDaVkCoBcFe@aAaFhBwLnAkDrHkIf@yAc@sEn@aPgCsXgAyC_AkIl@mHZsAbB}@fK{OrLiWoGaL]kXd@gAQwC[SImLqCVmD|Bw@dD_EpDqUcAyAgAmAGiHFqKq@sCFuFaDqAj@uBfC}AVo@[e@ZcAo@yAFuGcBuGvC{CdCeDbH{@NkHuGaDc@eBJaJzMe@dWgChIR`K}FpN_DRcC_AqVaNPuK"
                }
            ]
        },
        "MediaItems": [
            {
                "BaseUri": "https://ms.immowelt.org",
                "FileName": "ec7ea2ab-4ea0-494b-babf-8da4a08e3fe9__94193.jpg",
                "FileSize": 223954,
                "Location": "https://ms.immowelt.org/56094624-23c5-4ded-9aaa-a3b74850a63e/e273898e-9cf6-41d1-98b0-0595f86b50b6",
                "MediaId": "56094624-23c5-4ded-9aaa-a3b74850a63e",
                "MimeType": "image/jpeg",
                "Position": 0,
                "StoragePath": "/56094624-23c5-4ded-9aaa-a3b74850a63e/e273898e-9cf6-41d1-98b0-0595f86b50b6",
                "Title": "Visualisierung"
            },
            {
                "BaseUri": "https://ms.immowelt.org",
                "FileName": "d3ee83b9-5ef5-45d1-ab31-12ac87dd0f6a__94189.jpg",
                "FileSize": 2104607,
                "Location": "https://ms.immowelt.org/78b40967-c4c7-4af9-ae05-66ced952a1b8/0e444f51-795c-4241-8564-2b1b1972078d",
                "MediaId": "78b40967-c4c7-4af9-ae05-66ced952a1b8",
                "MimeType": "image/jpeg",
                "Position": 1,
                "StoragePath": "/78b40967-c4c7-4af9-ae05-66ced952a1b8/0e444f51-795c-4241-8564-2b1b1972078d",
                "Title": "Visualisierung"
            },
            {
                "BaseUri": "https://ms.immowelt.org",
                "FileName": "a2158f83-8ff1-4c4f-aabe-3346c5411733__94195.jpg",
                "FileSize": 335528,
                "Location": "https://ms.immowelt.org/673d7f44-6447-446a-a92c-61a919ac4654/b755630d-13be-4cc0-8741-94b97e096598",
                "MediaId": "673d7f44-6447-446a-a92c-61a919ac4654",
                "MimeType": "image/jpeg",
                "Position": 2,
                "StoragePath": "/673d7f44-6447-446a-a92c-61a919ac4654/b755630d-13be-4cc0-8741-94b97e096598",
                "Title": "Lageplan"
            },
            {
                "BaseUri": "https://ms.immowelt.org",
                "FileName": "a9b788fe-6fc3-4b42-a474-986298864ba3__94197.jpg",
                "FileSize": 369197,
                "Location": "https://ms.immowelt.org/4d72d652-8af8-4e2c-82ef-00156f2ab6bd/b5b9d8e1-1123-47a3-9562-30dffccffe17",
                "MediaId": "4d72d652-8af8-4e2c-82ef-00156f2ab6bd",
                "MimeType": "image/jpeg",
                "Position": 3,
                "StoragePath": "/4d72d652-8af8-4e2c-82ef-00156f2ab6bd/b5b9d8e1-1123-47a3-9562-30dffccffe17",
                "Title": "Lageplan"
            },
            {
                "BaseUri": "https://ms.immowelt.org",
                "FileName": "06cdce48-2973-4c2e-b51c-2e813a9350f4__94199.jpg",
                "FileSize": 284088,
                "Location": "https://ms.immowelt.org/6f6fc8d9-3ae5-4244-9ef5-6ed60137a7e0/7d7ea56d-56ad-4563-8e2a-ae7b5d522f04",
                "MediaId": "6f6fc8d9-3ae5-4244-9ef5-6ed60137a7e0",
                "MimeType": "image/jpeg",
                "Position": 4,
                "StoragePath": "/6f6fc8d9-3ae5-4244-9ef5-6ed60137a7e0/7d7ea56d-56ad-4563-8e2a-ae7b5d522f04",
                "Title": "Freifl\u00e4chenplan"
            },
            {
                "BaseUri": "https://ms.immowelt.org",
                "FileName": "5b325dba-4e31-468d-ba68-25dbdfacc20f__94201.jpg",
                "FileSize": 216346,
                "Location": "https://ms.immowelt.org/0b38a06a-9783-4576-a0b4-8c82a1a33a06/c41efe0e-3024-4f17-8bd0-7e9af6635816",
                "MediaId": "0b38a06a-9783-4576-a0b4-8c82a1a33a06",
                "MimeType": "image/jpeg",
                "Position": 5,
                "StoragePath": "/0b38a06a-9783-4576-a0b4-8c82a1a33a06/c41efe0e-3024-4f17-8bd0-7e9af6635816",
                "Title": "Grundriss TG"
            },
            {
                "BaseUri": "https://ms.immowelt.org",
                "FileName": "a82b73b6-7027-484c-8cad-2eef78ebe939__94203.jpg",
                "FileSize": 92653,
                "Location": "https://ms.immowelt.org/7dcc2481-8f60-45b0-8e28-5fef7479a7be/f3a7fb81-eedf-418e-b687-64cc633fcfbf",
                "MediaId": "7dcc2481-8f60-45b0-8e28-5fef7479a7be",
                "MimeType": "image/jpeg",
                "Position": 6,
                "StoragePath": "/7dcc2481-8f60-45b0-8e28-5fef7479a7be/f3a7fb81-eedf-418e-b687-64cc633fcfbf",
                "Title": "Wohnung Nr. 6"
            }
        ],
        "Documents": [],
        "OnlineId": "2CWC457",
        "Visible": true,
        "MediaId": 0,
        "Offerer": {
            "addressGuid": "B6CD62EE9FF644D19A24AE531A0DE223",
            "globalUserId": 56453,
            "sellerType": "partner",
            "projectId": "iww",
            "brokerValuation": {
                "ReviewsEnabled": true,
                "KeyStatistics": {
                    "TotalReviewCount": 0,
                    "AverageOverallRating": 0
                }
            },
            "settings": {
                "exposeHyperlinksAllowed": false,
                "officeHours": ""
            },
            "contactData": {
                "companyName": "I\u00b4M LIVING Immobilien",
                "address": {
                    "city": "M\u00fcnchen",
                    "zipCode": "80539",
                    "street": "Maximilianstr. 40"
                },
                "profilePicture": "https://filestore.immowelt.de/ProfilBilder/150_526c9677a54740bbafcb531333130fc2.jpg",
                "salutation": "Frau",
                "firstName": "Christine",
                "lastName": "Burmeister",
                "position": "Selbst\u00e4ndige Immobilienmaklerin",
                "mobile": "0160 / 187 86 08",
                "phone": "089 / 121 90 680"
            },
            "immoweltExclusive": {
                "exclusivityEnabled": false
            },
            "links": [
                {
                    "Url": "https://www.immowelt.de/profil/56453",
                    "ExternalLink": false,
                    "LinkType": "BROKERPROFILE"
                },
                {
                    "Url": "https://www.immowelt.de/profil/56453",
                    "ExternalLink": false,
                    "LinkType": "COMPANYPROFILE"
                },
                {
                    "Url": "https://www.immowelt.de/profil/56453#objects",
                    "ExternalLink": false,
                    "LinkType": "OBJECTS"
                },
                {
                    "Url": "https://www.immowelt.de/profil/56453#reviews",
                    "ExternalLink": false,
                    "LinkType": "REVIEWS"
                },
                {
                    "Url": "https://www.immowelt.de/profil/56453#impressum",
                    "ExternalLink": false,
                    "LinkType": "IMPRESSUM"
                },
                {
                    "ExternalLink": true,
                    "Url": "http://www.i-m-living.de/",
                    "LinkType": "HOMEPAGE"
                }
            ],
            "logoUrl": "https://filestore.immowelt.de/logo/left_7dcf25d553bf43b4a6592bddc0be2beb.jpg"
        },
        "Project": {
            "Address": {
                "City": "M\u00fcnchen",
                "CountryCode": "DE",
                "PostCode": "80999",
                "Street": "Theodor-Fischer-Strasse",
                "HouseNumber": "99"
            },
            "AreaRange": {
                "Max": 84,
                "Min": 55
            },
            "ContactPersonGuid": 5159148,
            "CreateDate": "2023-11-09T16:38:26.480Z",
            "EstateObjectLinks": [
                {
                    "GlobalObjectKey": "faacebad-82eb-4987-89ee-26feee19f01f"
                },
                {
                    "GlobalObjectKey": "6f1faa75-1228-4b40-ba6f-955ade7bb4e0"
                },
                {
                    "GlobalObjectKey": "a10f3a30-336f-44a6-88d6-2b2fba8627a1"
                },
                {
                    "GlobalObjectKey": "01c0c334-fd75-442a-bc86-83894df68d0a"
                },
                {
                    "GlobalObjectKey": "ef89aa6a-58a2-4dd9-9720-269e54a1bd11"
                },
                {
                    "GlobalObjectKey": "cb68ca03-f2a0-40ef-9479-ec7bcd3c839d"
                }
            ],
            "EstateObjectsCount": 8,
            "Guid": 56453,
            "Id": 6002,
            "OnlineId": "k2kv732",
            "PriceRange": {
                "Currency": "EUR",
                "Min": 599000,
                "Max": 955000
            },
            "ProjectDescription": [
                {
                    "Header": "Projektbeschreibung",
                    "Text": "Diese moderne und energetisch auf dem neusten Stand geplante Wohnung, wird in einem 8-Parteienhaus auf einem 799 m\u00b2 gro\u00dfen Grundst\u00fcck erstellt. Die gesamte Anlage wird bis Ende 2024 erbaut.\n\nDas Haus entsteht in massiver Bauweise und weist eine hochwertige Ausstattung auf. Die Wohnung wird schl\u00fcsselfertig, inklusiver der Au\u00dfenanlagen und Bepflanzung \u00fcbergeben.\n\nNehmen Sie alle Annehmlichkeiten eines Neubaus in Anspruch - wie Zahlung nach Baufortschritt - neuste Technik - Gew\u00e4hrleistung!\n\nIn diesem Haus stehen noch weitere Wohnungen zum Kauf zur Verf\u00fcgung.\n\nGerne unterst\u00fctzen wir Sie auch bei der Auswahl einer Finanzierung.\n\nDetails der Ausstattung:\n\n- PV Anlage zur Stromerzeugung f\u00fcr W\u00e4rmepumpe \n einschl. Batteriespeicher\n- Ziegelbauweise mit W\u00e4rmed\u00e4mmung\n- Elektrische Rolll\u00e4den mit Zeitschaltuhr\n- Fu\u00dfbodenheizung in allen Wohn- und Hobbyr\u00e4umen\n- Eiche-Parkettb\u00f6den\n- 3-fach Isolierverglasung\n- Beheizung mit moderner W\u00e4rmepumpe der Firma Viessmann \noder gleichwertig.\n- Bodengleiche Duschen mit Echtglas-Duschtrennw\u00e4nden\n- Baubegleitendes T\u00dcV Baucontrolling mit Dokumentation\n- Dezentrales Be- und Entl\u00fcftungssystem \n mit W\u00e4rmer\u00fcckgewinnung\n- Umfangreiche Elektroausstattung mit CAT-7 EDV Verkabelung\n\nWeitere Informationen stellen wir Ihnen gerne anhand der Baubeschreibung zur Verf\u00fcgung.\n\nGerne stehen wir f\u00fcr ein pers\u00f6nliches Gespr\u00e4ch oder einen Besichtigungstermin zur Verf\u00fcgung - wir freuen uns auf Ihre Kontaktaufnahme!"
                }
            ],
            "ProjectLogoLink": "https://ms.immowelt.org/9ce39a27-0e53-4ffb-b115-437a7ae681e0/50fcce08-3fdf-4e1b-9487-7f5a25be40d0",
            "ProjectStatus": {
                "ConstructionStartDate": "2023",
                "ConstructionEndDate": "2024",
                "MovingDate": "2024"
            },
            "RoomRange": {
                "Max": 3,
                "Min": 2
            },
            "Title": "Energieeffizente Wohnungen zum Eigennutz oder als Kapitalanlage"
        }
    }
]    

Our immowelt.de scraper can successfully scrape property pages. Next, let's add
search page scraping to help us discover the properties we want to scrape.

How to Scrape Immowelt.de Search Pages?

Before we start scraping immowelt.de search pages, let's take a look at what the search pages look like. Search for any properties on the website and you will get a page similar to this:

search page on immowelt.de
Search page on immowelt.de

We used the following URL to get the above search results:

https://www.immowelt.de/suche/muenchen/wohnungen/kaufen?sort=relevanz

To scrape search pages, we'll use the search API to get the data directly in JSON. Imoowelt has a hidden web API that is called when search requests are being made.

How to Scrape Hidden APIs

For a full introduction to hidden API scraping see this tutorial article which covers this subject in greater detail

How to Scrape Hidden APIs

To scrape this, we need to find the API responsible for fetching the search data. To do that, follow the steps below:

  1. Open developer tools by clicking the F12 key.
  2. Head over to the Network tab and filter requests by clicking the Fetch/XHR button.
  3. Fetch more search data by clicking on the search page button.

By following the above steps, you should see all the requests sent from the browser to the search API:

scrapfly middleware
Background API requests

We can see different XHR requests but we are only in the searches request. To see the response of this request, select it and click on the preview button:

search API response
Search API response

We can also view the headers and body of the request by clicking on the Headers and Payload buttons:

request headers and body
Request headers and body

The request body controls the search results, while the request headers represent the headers required by the server to authorize and validate the request.

To scrape this search API data, we'll replicate this request within our scraper by using the same request headers and payload. First, we'll start by setting up an httpx client with the headers and body same as the search API request:

Python
ScrapFly
# Add the authorization token below by getting a new one from developer tools
auth_token = ""

async def send_api_request(page_number: int):
    """get search data from the search API"""
    # add the request headers
    headers = {
        "accept": "application/json, text/plain, */*",
        "accept-language": "en-US,en;q=0.9",
        # Add the authorization token below by getting a new one from developer tools as it may get expired
        "authorization": f"Bearer {auth_token}",
        "content-type": "application/json",
    }
    # add the request payload
    data = {
        "estateGroups": ["RESIDENTIAL", "COMMERCIAL"],
        "estateType": "APARTMENT",
        "distributionTypes": ["SALE"],
        "estateSubtypes": [],
        # locations ids of the search location, change them to search within a specific location
        "locationIds": [4916],
        # narrow down the search results by adding more search fields
        "primaryPrice": {"min": None, "max": None},
        "primaryArea": {"min": None, "max": None},
        "areas": [{"areaType": "PLOT_AREA", "min": None, "max": None}],
        "rooms": {"min": None, "max": None},
        "constructionYear": {"min": None, "max": None},
        "geoRadius": {"radius": None, "point": {"lat": None, "lon": None}},
        "sort": {"direction": "DESC", "field": "RELEVANCE"},
        "immoItemTypes": ["ESTATE", "PROJECT"],
        "paging": {"size": 20, "page": page_number},
    }
    # create a new httpx client with new headers and payload values
    client = AsyncClient(headers=headers)
    # send a POST request to the search API
    response = await client.post(
        url="https://api.immowelt.com/residentialsearch/v1/searches",
        data=json.dumps(data)
    )
    return response    
# Add the authorization token below by getting a new one from developer tools
auth_token = ""

async def send_api_request(page_number: int):
    """get search data from the search API"""
    # add the request headers
    headers = {
        "accept": "application/json, text/plain, */*",
        "accept-language": "en-US,en;q=0.9",
        # Add the authorization token below by getting a new one from developer tools as it may get expired
        "authorization": f"Bearer {auth_token}",
        "content-type": "application/json",
    }
    # add the request payload
    data = {
        "estateGroups": ["RESIDENTIAL", "COMMERCIAL"],
        "estateType": "APARTMENT",
        "distributionTypes": ["SALE"],
        "estateSubtypes": [],
        # locations ids of the search location, change them to search within a specific location
        "locationIds": [4916],
        # narrow down the search results by adding more search fields
        "primaryPrice": {"min": None, "max": None},
        "primaryArea": {"min": None, "max": None},
        "areas": [{"areaType": "PLOT_AREA", "min": None, "max": None}],
        "rooms": {"min": None, "max": None},
        "constructionYear": {"min": None, "max": None},
        "geoRadius": {"radius": None, "point": {"lat": None, "lon": None}},
        "sort": {"direction": "DESC", "field": "RELEVANCE"},
        "immoItemTypes": ["ESTATE", "PROJECT"],
        "paging": {"size": 20, "page": page_number},
    }

    # send a POST request to the search API
    response = await scrapfly.async_scrape(ScrapeConfig(
        url="https://api.immowelt.com/residentialsearch/v1/searches",
        headers=headers,
        body=json.dumps(data),
        # set the proxy country to Germany
        country="DE",
        method="POST"
    ))
    return response    

Here, we define a send_api_request function to send a POST requests to the immowelt.de search API. Next, we'll use this function to crawl over search pages and parse the JSON response:

Python
ScrapFly
import asyncio
import json
from typing import List, Dict
from httpx import AsyncClient, Response

# Add the authorization token below by getting a new one from developer tools as it may get expired
auth_token = ""

async def send_api_request(page_number: int):
    # the remaining of the function logic here

def parse_search_api(response: Response) -> List[Dict]:
    """parse the search API response"""
    data = json.loads(response.text)
    search_data = data["data"]
    max_search_pages = data["pagesCount"]
    return {
        "search_data": search_data,
        "max_search_pages": max_search_pages
    }  

async def scrape_search(scrape_all_pages: bool, max_scrape_pages: int) -> List[Dict]:
    """scrape search pages usign the search API"""
    # scrape the first search page
    first_page = await send_api_request(1)
    data = parse_search_api(first_page)
    search_data = data["search_data"]
    # get the the number of available search pages
    max_search_pages = data["max_search_pages"]
    # scrape all available pages in the search if scrape_all_pages = True or max_pages > total_search_pages        
    if scrape_all_pages == False and max_scrape_pages < max_search_pages:
        max_scrape_pages = max_scrape_pages
    else:
        max_scrape_pages = max_search_pages
    # scrape the remaining search pages from the API
    for page in range(2, max_scrape_pages + 1):
        response = await send_api_request(page)
        data = parse_search_api(response)["search_data"]
        search_data.extend(data)
    print(f"scraped {len(search_data)} properties from search")
    return search_data    
import json
import asyncio
from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse
from typing import Dict, List

scrapfly = ScrapflyClient(key="Your ScrapFly API key")

# Add the authorization token below by getting a new one from developer tools
auth_token = ""

async def send_api_request(page_number: int):
    # the remaining of the function logic here

def parse_search_api(response: ScrapeApiResponse) -> List[Dict]:
    """parse the search API response"""
    data = json.loads(response.scrape_result["content"])
    search_data = data["data"]
    max_search_pages = data["pagesCount"]
    return {
        "search_data": search_data,
        "max_search_pages": max_search_pages
    }

async def scrape_search(scrape_all_pages: bool, max_scrape_pages: int) -> List[Dict]:
    """scrape search pages usign the search API"""
    # scrape the first search page
    first_page = await send_api_request(1)
    data = parse_search_api(first_page)
    search_data = data["search_data"]
    # get the the number of available search pages
    max_search_pages = data["max_search_pages"]
    # scrape all available pages in the search if scrape_all_pages = True or max_pages > total_search_pages 
    if scrape_all_pages == False and max_scrape_pages < max_search_pages:
        max_scrape_pages = max_scrape_pages
    else:
        max_scrape_pages = max_search_pages
    # scrape the remaining search pages from the API
    for page_number in range(2, max_scrape_pages + 1):
        response = await send_api_request(page_number)
        data = parse_search_api(response)["search_data"]
        search_data.extend(data)
    print(f"scraped {len(search_data)} properties from search")
    return search_data    
Run the code
if __name__ == "__main__":
    search_data = asyncio.run(scrape_search(
        scrape_all_pages=False,
        max_scrape_pages=3
    ))
    # print the result in JSON format
    print(json.dumps(search_data, indent=2))    

The result is a list containing data from three search API pages:

Sample output
[
  {
    "id": "project_4401",
    "itemType": "PROJECT",
    "projectId": "4401",
    "locationIds": [
      529709,
      529492,
      528418,
      464967,
      4916,
      115926,
      122156,
      121726
    ],
    "place": {
      "city": "München"
    },
    "estateTypes": [
      "APARTMENT"
    ],
    "estateSubtypes": [],
    "distributionType": "SALE",
    "title": " An der Rehwiese 3 & 9 - München",
    "prices": [
      {
        "amountMin": 709600.0,
        "amountMax": 1024500.0,
        "type": "PURCHASE_PRICE",
        "currency": "EUR",
        "attributes": []
      }
    ],
    "areas": [
      {
        "sizeMin": 55.0,
        "sizeMax": 114.0,
        "type": "LIVING_AREA"
      }
    ],
    "primaryArea": {
      "sizeMin": 55.0,
      "sizeMax": 114.0,
      "type": "LIVING_AREA"
    },
    "primaryPrice": {
      "amountMin": 709600.0,
      "amountMax": 1024500.0,
      "type": "PURCHASE_PRICE",
      "currency": "EUR",
      "attributes": []
    },
    "roomsMin": 2.0,
    "roomsMax": 4.0,
    "timestamp": "2022-10-31T12:23:08.502Z",
    "isNew": false,
    "features": [
      "NEW_BUILDING",
      "NEW_BUILDING_PROJECT"
    ],
    "pictures": [
      {
        "imageUri": "https://ms.immowelt.org/4c985c53-c27d-4dc0-9eaf-12b9cf7ffef8/7fcce537-d7ee-4212-8602-c95e3a24d531",
        "imageUriBasePath": "https://ms.immowelt.org/4c985c53-c27d-4dc0-9eaf-12b9cf7ffef8/7fcce537-d7ee-4212-8602-c95e3a24d531"
      },
      {
        "imageUri": "https://ms.immowelt.org/9b9243cb-d7c7-45c4-bfd7-3afdcf43a555/74cda368-a126-4321-b868-180fba731af9",
        "imageUriBasePath": "https://ms.immowelt.org/9b9243cb-d7c7-45c4-bfd7-3afdcf43a555/74cda368-a126-4321-b868-180fba731af9"
      }
    ],
    "presentationProducts": [],
    "broker": {
      "guid": 28000,
      "sellerType": "PARTNER",
      "companyName": "Stauch Wohnbau GmbH",
      "partnerAward": "PLATIN",
      "immoweltExclusive": false,
      "logoUri": "//filestore.immowelt.de/logo/100_afa77f7be8254db1b48ff73941b6d194.jpg",
      "logoUriHttps": "https://filestore.immowelt.de/logo/100_afa77f7be8254db1b48ff73941b6d194.jpg"
    },
    "onlineId": "k2jb632",
    "projectData": {
      "constructionStartDate": "2022",
      "constructionEndDate": "2024",
      "movingDate": "2024",
      "logoUri": "https://ms.immowelt.org/516ff6e5-b724-4f3f-8232-07c7280172db/95a8a292-d10b-4048-b619-3991ccc0e769",
      "numberOfUnits": 18
    },
    "relatedItems": [
      {
        "itemType": "ESTATE",
        "onlineId": "2ble55s",
        "gok": "f7c6c447-aacd-46b0-bb44-9cb8b5b5fd97",
        "projectId": null,
        "title": null,
        "roomsMin": 3.5,
        "roomsMax": 3.5,
        "primaryPrice": {
          "amountMin": 939200.0,
          "amountMax": 939200.0,
          "type": "PURCHASE_PRICE",
          "currency": "EUR",
          "attributes": []
        },
        "primaryArea": {
          "sizeMin": 99.0,
          "sizeMax": 99.0,
          "type": "LIVING_AREA"
        },
        "features": [
          "GARDEN_SHARED",
          "ATTIC",
          "CELLAR_SHARE",
          "BATH_WITH_TUB",
          "BATH_WITH_WINDOW",
          "PASSENGER_LIFT",
          "TERRACE",
          "UNDERGROUND_PARKING",
          "PARKING_AREA",
          "FREE_OF_COMMISSION"
        ]
      }
    ],
    "estateGroups": [
      "RESIDENTIAL"
    ]
  }
]   

Our immowelt.de scraper can successfully scrape property and search pages. However, we may start getting blocked after sending additional requests. Let's look at a solution!

Bypass Immowelt.de Scraping Blocking

To avoid scraping blocking on immowelt.de, we'll use ScrapFly. A web scraping API that scales up web scraper by providing:

scrapfly middleware
ScrapFly service does the heavy lifting for you

Here is how to use the asp feature with the ScrapFly Python SDK to scrape immowelt.de without getting blocked:

import httpx
from parsel import Selector

response = httpx.get("some immowelt.de url")
selector = Selector(response.text)

# in ScrapFly SDK becomes
from scrapfly import ScrapeConfig, ScrapflyClient, ScrapeApiResponse

scrapfly_client = ScrapflyClient("Your ScrapFly API key")
result: ScrapeApiResponse = scrapfly_client.scrape(ScrapeConfig(
    # some immowelt.de URL
    url="https://www.immowelt.de/expose/2czpn55",
    # select a specific proxy country
    country="DE",
    # enable anti scraping protection bypass
    asp=True,
    # allows JavaScript rendering similar to headless browsers
    render_js=True
))
# use the built-in parsel selector
selector = result.selector

FAQ

To wrap up this guide, let's take a look at some frequently asked questions about web scraping immowelt.de.

Yes, all the data on immowelt.de is publicly available and it's legal to scrape them as long as you follow the website's terms of service and keep your scraping rate reasonable. For further details, refer to our previous guide - is web scraping legal.

Is there a public API for immowelt.de?

At the time of writing, there is no public API for immowelt.de. However, we have seen that's possible to use the private immowelt.de search API to scrape real estate data.

Are there alternatives for immowelt.de?

Yes, immobilienscout24.de is another popular website for real estate ads in Germany. We have covered web scraping immobilienscout24.de in a previous article. Refer to the #realestate scraping tag for more real estate property scraping targets.

How to Scrape Immobilienscout24.de Real Estate Data

Another popular target for real estate property scraping in Germany is Immobilienscout24.de and we have a guide for it as well!

How to Scrape Immobilienscout24.de Real Estate Data
Latest Immowelt.de Scraper Code
https://github.com/scrapfly/scrapfly-scrapers/

Web Scraping Immowelt.de Summary

In this article, we explained how to scrape immowelt.de with Python - a popular website for real estate ads in Germany.

We went through a step-by-step guide on scraping immowelt.de real estate data from property pages and search API. Furthermore, we explained how to avoid immowelt.de web scraping blocking.

Related Posts

How to Scrape Bing Search with Python

In this scrape guide we'll be taking a look at scraping Bing search results. It's the second biggest search engine in the world and it contains a lot of data - all retrievable with a bit a of Python.

How to Scrape G2 Company Data and Reviews

In this scrapeguide we're taking a look at G2.com - one of the biggest digital product metawebsites out there. We'll be scraping product data, reviews and company profiles.

How to Scrape Etsy.com Product, Shop and Search Data

In this scrapeguide we're taking a look at Etsy.com - a popular e-commerce market for hand crafted and vintage items. We'll be using Python and HTML parsing to scrape search and product data.