     [Blog](https://scrapfly.io/blog)   /  [crawling](https://scrapfly.io/blog/tag/crawling)   /  [How to Find All URLs on a Domain](https://scrapfly.io/blog/posts/how-to-find-all-urls-on-a-domain)   

# How to Find All URLs on a Domain

 by [Ziad Shamndy](https://scrapfly.io/blog/author/ziad) Nov 17, 2026 25 min read [\#crawling](https://scrapfly.io/blog/tag/crawling) [\#python](https://scrapfly.io/blog/tag/python) 

 [  ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-find-all-urls-on-a-domain "Share on LinkedIn") [  ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-find-all-urls-on-a-domain&text=How%20to%20Find%20All%20URLs%20on%20a%20Domain "Share on X") [  ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-find-all-urls-on-a-domain "Share on Facebook")    

 

 

Summarize this article with

 [  ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-find-all-urls-on-a-domain) [  ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-find-all-urls-on-a-domain) [  ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-find-all-urls-on-a-domain) [  ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-find-all-urls-on-a-domain) [  ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-find-all-urls-on-a-domain) 



      

Usually, to find all pages on a website we write a web crawler that starts at the homepage and follows every internal link. However, a crawler finishing with zero errors doesn't mean we found every URL on the domain. When we tested a link crawler on web-scraping.dev, our script scraped every link on the homepage without issues, though it missed all 28 product pages listed in the XML sitemap. It stopped at 10 pages, and products 26 to 28 are only linked from a related-products section that changes on every load.

In this article, we'll explain how to find all URLs on a domain by combining sitemaps, link crawlers, and web archives in Python. We'll start by discovering XML sitemaps from robots.txt, and then we'll build a Python crawler with httpx and parsel, and another with Scrapy. Finally, we'll normalize our URLs and merge all sources into one clean dataset. Let's get started!



## Key Takeaways

In this article, we've taken a look at a few different ways to find all URLs on a domain. Here's what we learned:

- Sitemaps are the quickest way to get an overview of the website structure. We found two of them through the `/robots.txt` file of `web-scraping.dev` and they listed 5,038 URLs.
- Not every website supports sitemaps though and they only list what the website owner put in them. One of the URLs we found there, `/sitemap-404-target`, didn't even exist.
- Then, we built a web crawler with Python alongside with a few Python packages. It explores every link on the website and without a page limit it found everything from `/sitemap.xml` within 162 pages.
- Crawlers need limits like a page count or crawling depth as some pages like calendars never run out of links.
- Scrapy's `CrawlSpider` can do the same, just don't forget to enable `ROBOTSTXT_OBEY` when running it from a script.
- Pages that aren't linked anywhere can sometimes be found through search engines or the Wayback Machine.
- For bigger websites, Scrapfly's [Crawler API](https://scrapfly.io/products/crawler-api) can run the whole crawl for you.

**Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime.







## Where to Find All URLs of a Domain?

If we own the website, the database or server logs have every URL. For a website we don't own, we have to combine a few public sources.

Sitemaps are web files that list the locations of web objects like product pages, articles etc. for web crawlers.

It's mostly used to tell search engines what to index though in web scraping we can use them to discover targets to scrape.

On `web-scraping.dev`, the `/sitemap.xml` file lists all 28 product pages. The five `/products` pages only show products 1 to 25 though.

Most commonly, web crawlers are used to discover targets when the website doesn't have a target directory or a sitemap.

For example, if an e-commerce website doesn't have a product directory we could crawl all of its web pages and find all of the products through backlinks like the "related products" section and so on.

That's how products 26, 27 and 28 show up on `web-scraping.dev`. Outside the sitemap, we only found them in the related products section, and it changes on every request.

Alternatively, we can take advantage of public search indexes such as google.com or bing.com using queries like `site:web-scraping.dev`.

The Wayback Machine has an index too. We can query it through its CDX API with a single request:

shell

```shell
curl "https://web.archive.org/cdx/search/cdx?url=web-scraping.dev/*&fl=original,timestamp,statuscode&collapse=urlkey"
```



It returned 2,632 lines for us, here's a few of them:

text

```text
https://web-scraping.dev/%5C 20260930203227 404
https://web-scraping.dev/crawler-test-report 20260413112552 200
https://web-scraping.dev/js-links 20260413113651 200
https://web-scraping.dev/product/26 20250212005928 200
https://web-scraping.dev/products&quot; 20250917201727 404
https://web-scraping.dev/session-vortex 20260413120758 200
```



`/js-links` and `/session-vortex` aren't in the sitemaps and the homepage doesn't link them, but they still load. The `%5C` and `&quot;` ones are just broken captures.

If you only need the links of a single page, see our guide on how to [extract every link from one page](https://scrapfly.io/blog/answers/how-to-find-all-links-using-beautifulsoup). Otherwise, let's start with the sitemaps since they take the fewest requests.



## Where to Find Sitemaps?

Scraping sitemaps is an efficient way to discover thousands of organized URLs. They are provided for search engine crawlers to index the pages, which we can use to discover web scraping targets on a website.

Sitemaps are usually located at the root of the website `/sitemap.xml` file. However, there's no clear standard location so another way to find the sitemap location is to explore the standard `/robots.txt` file which provides instructions for web crawlers.

For example, the `web-scraping.dev/robots.txt` file contains the following lines:

http

```http
User-agent: *
Disallow: /robots-disallowed
Crawl-delay: 2

User-agent: Googlebot
Disallow: /api/
Disallow: /robots-disallowed
Crawl-delay: 2

User-agent: Bingbot
Disallow: /api/
Disallow: /robots-disallowed
Crawl-delay: 2

Sitemap: https://web-scraping.dev/sitemap.xml
Sitemap: https://web-scraping.dev/sitemap-index.xml
```



We only need the two `Sitemap:` lines at the bottom. The `Disallow` rules are just crawling preferences, they don't make a page private and don't tell us if it exists.

Many programming languages have robot.txt parsers built-in or it's as easy as retrieving the page content and finding the line with `Sitemap:` prefix. In Python, `RobotFileParser` gives us the list right away:

python

```python
from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://web-scraping.dev/robots.txt")
rp.read()
print(rp.site_maps())
```



text

```text
['https://web-scraping.dev/sitemap.xml', 'https://web-scraping.dev/sitemap-index.xml']
```



Sitemaps are XML documents with specific, simple structures. When it comes to web scraping we're mostly interested in the `<urlset>` values where each `<url>` contains location and some optional metadata.

For bigger websites, the sitemap limit of 50 000 URLs is often not enough so there are multiple sitemap files contained in a sitemap hub. To scrape this we need a bit of recursion.

The second sitemap on `web-scraping.dev` is one of these hubs:

xml

```xml
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
  <loc>https://web-scraping.dev/sitemap-1.xml</loc>
</sitemap>
<sitemap>
  <loc>https://web-scraping.dev/sitemap-2.xml</loc>
</sitemap>
...
</sitemapindex>
```



It goes all the way to `sitemap-50.xml`.

Sometimes, sitemap hubs split sitemaps by category, location which can help us to target specific types of pages. Though to scrape hubs all we have to do is parse the index file and scrape each sitemap file:

python

```python
from urllib.robotparser import RobotFileParser
from parsel import Selector
import httpx

rp = RobotFileParser("https://web-scraping.dev/robots.txt")
rp.read()
sitemaps = rp.site_maps() or ["https://web-scraping.dev/sitemap.xml"]

urls = []
for sitemap_url in sitemaps:
    response = httpx.get(sitemap_url)
    selector = Selector(response.text)
    selector.remove_namespaces()
    # sitemap hubs list other sitemap files
    for child_url in selector.xpath("//sitemap/loc/text()").getall():
        response = httpx.get(child_url)
        child = Selector(response.text)
        child.remove_namespaces()
        urls.extend(child.xpath("//url/loc/text()").getall())
    # regular sitemaps list page urls
    urls.extend(selector.xpath("//url/loc/text()").getall())

print(f"found {len(urls)} urls")
for url in urls[:3]:
    print(url)
```



Running it against `web-scraping.dev` took about 40 seconds and here is what we got:

text

```text
found 5038 urls
https://web-scraping.dev/products
https://web-scraping.dev/product/1
https://web-scraping.dev/product/2
```



That's 37 URLs from `/sitemap.xml` and 5,001 from the 50 child sitemaps.

We left the `remove_namespaces()` calls out at first and got this:

text

```text
found 0 urls
```



That's the sitemap namespace getting in the way of `//url`.

With a single `httpx.Client()` instead of `httpx.get()` it took 18 seconds for us.

When scraping links gathered from sitemaps, robots.txt and other directories that are rarely used by human users but visible to bots it's best to validate the existence of these pages on the live website.

One of the URLs in `sitemap-1.xml` is `/sitemap-404-target`, and here's what we get when we request it:

shell

```shell
$ curl -s -o /dev/null -w "%{http_code}\n" https://web-scraping.dev/sitemap-404-target
404
```



For gzip compressed sitemaps and more examples, see our [scrape sitemaps](https://scrapfly.io/blog/posts/how-to-scrape-sitemaps) guide.

[How to Scrape Sitemaps to Discover Scraping TargetsUsually to find scrape targets we look at site search or category pages but there's a better way - sitemaps! In this tutorial, we'll be taking a look at how to find and scrape sitemaps for target locations.](https://scrapfly.io/blog/posts/how-to-scrape-sitemaps)

Sitemaps only have what the website owner put in them though. So, let's take a look at how to crawl the website itself next.



## Crawling a Domain With Python

The most important component of a web crawler is its exploration mechanism, which introduces many new components like URL discovery and filtering.

The crawler starts with a pool of URLs (the initial seed is often called *start urls*) and scrapes their responses (HTML data).

Then, the responses are **parsed** for more URLs to follow which are being filtered and added to the next crawl loop pool.

We can easily extract all URLs from an HTML document by extracting the `href` attributes of all `<a>` nodes. For this, we can use `parsel` package with either [CSS selectors](https://scrapfly.io/blog/posts/parsing-html-with-css#what-are-css-selectors) or [XPath selectors](https://scrapfly.io/blog/posts/parsing-html-with-xpath#what-is-xpath).

A plain URL extractor like this is very primitive and we can't use it in our crawler as it produces duplicate and non-crawlable urls (like downloadable files). So, we also need a filter that can:

- Normalize found URLs and deduplicate them.
- Filter out offsite URLs (of a different domain like social media links etc.)
- Filter out uncrawlable URLs (like file download links)
- Skip URLs that `robots.txt` disallows.

For this, we'll be using `w3lib` and `tldextract` libraries which offer great utility functions for processing URLs. These Python packages can be installed through `pip install` console command:

shell

```shell
pip install httpx parsel w3lib tldextract loguru
```



This is the crawler from our [crawling with Python](https://scrapfly.io/blog/posts/crawling-with-python) guide, with a page limit and a `robots.txt` check added:

python

```python
import asyncio
import posixpath
import sys
from typing import List
from urllib.parse import urljoin, urlparse
from urllib.robotparser import RobotFileParser

import httpx
from parsel import Selector
from tldextract import tldextract
from w3lib.url import canonicalize_url
from loguru import logger as log


class UrlFilter:
    IGNORED_EXTENSIONS = [
        # archives
        '.7z', '.rar', '.tar', '.gz', '.zip',
        # images
        '.gif', '.jpg', '.jpeg', '.png', '.svg', '.ico', '.webp',
        # other
        '.css', '.js', '.json', '.xml', '.pdf',
    ]

    def __init__(self, domain: str, subdomain: str = "", robots: RobotFileParser = None) -> None:
        # restrict filtering to specific domain and subdomain
        self.domain = domain
        self.subdomain = subdomain
        self.robots = robots
        self.seen = set()

    def is_valid_ext(self, url):
        """ignore non-crawlable documents"""
        return posixpath.splitext(urlparse(url).path)[1].lower() not in self.IGNORED_EXTENSIONS

    def is_valid_scheme(self, url):
        """ignore non http/s links"""
        return urlparse(url).scheme in ['https', 'http']

    def is_valid_domain(self, url):
        """ignore offsite urls"""
        parsed = tldextract.extract(url)
        return parsed.top_domain_under_public_suffix == self.domain and parsed.subdomain == self.subdomain

    def is_allowed(self, url):
        """ignore urls disallowed by robots.txt"""
        return self.robots is None or self.robots.can_fetch("*", url)

    def is_new(self, url):
        """ignore visited urls (in canonical form)"""
        return canonicalize_url(url) not in self.seen

    def filter(self, urls: List[str]) -> List[str]:
        """filter list of urls"""
        found = []
        for url in urls:
            if not self.is_valid_scheme(url):
                log.debug(f"drop ignored scheme {url}")
                continue
            if not self.is_valid_domain(url):
                log.debug(f"drop domain missmatch {url}")
                continue
            if not self.is_valid_ext(url):
                log.debug(f"drop ignored extension {url}")
                continue
            if not self.is_allowed(url):
                log.debug(f"drop robots.txt disallowed {url}")
                continue
            if not self.is_new(url):
                log.debug(f"drop duplicate {url}")
                continue
            self.seen.add(canonicalize_url(url))
            found.append(url)
        return found


class Crawler:
    async def __aenter__(self):
        self.session = await httpx.AsyncClient(
            timeout=httpx.Timeout(60.0),
            limits=httpx.Limits(max_connections=2),
            headers={
                "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
                "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
                "accept-language": "en-US;en;q=0.9",
            },
        ).__aenter__()
        return self

    async def __aexit__(self, *args, **kwargs):
        await self.session.__aexit__(*args, **kwargs)

    def __init__(self, url_filter: UrlFilter, max_pages: int = 50) -> None:
        self.url_filter = url_filter
        self.max_pages = max_pages
        self.visited = {}

    def parse(self, responses: List[httpx.Response]) -> List[str]:
        """find valid urls in responses"""
        all_unique_urls = set()
        for response in responses:
            if "text/html" not in response.headers.get("content-type", ""):
                continue
            sel = Selector(text=response.text, base_url=str(response.url))
            _urls_in_response = set(
                urljoin(str(response.url), url.strip())
                for url in sel.xpath("//a/@href").getall()
            )
            all_unique_urls |= _urls_in_response

        urls_to_follow = self.url_filter.filter(all_unique_urls)
        log.info(f"found {len(urls_to_follow)} urls to follow (from total {len(all_unique_urls)})")
        return urls_to_follow

    async def scrape(self, urls: List[str]):
        """scrape urls and return their responses"""
        responses = []
        failures = []
        log.info(f"scraping {len(urls)} urls")
        tasks = [self.session.get(url, follow_redirects=True) for url in urls]
        for url, result in zip(urls, await asyncio.gather(*tasks, return_exceptions=True)):
            if isinstance(result, httpx.Response):
                responses.append(result)
                self.visited[url] = result.status_code
            else:
                failures.append(result)
                self.visited[url] = "failed"
        return responses, failures

    async def run(self, start_urls: List[str], max_depth=5) -> None:
        """crawl target to maximum depth, page limit or until no more urls are found"""
        url_pool = self.url_filter.filter(start_urls)
        depth = 0
        while url_pool and depth <= max_depth and len(self.visited) < self.max_pages:
            url_pool = url_pool[: self.max_pages - len(self.visited)]
            responses, failures = await self.scrape(url_pool)
            log.info(f"depth {depth}: scraped {len(responses)} pages and failed {len(failures)}")
            url_pool = self.parse(responses)
            depth += 1


async def run():
    robots = RobotFileParser("https://web-scraping.dev/robots.txt")
    robots.read()
    url_filter = UrlFilter(domain="web-scraping.dev", robots=robots)
    async with Crawler(url_filter, max_pages=50) as crawler:
        await crawler.run(["https://web-scraping.dev/"])
    for url, status in crawler.visited.items():
        print(status, url)


if __name__ == "__main__":
    # only show INFO logs, DEBUG shows every dropped url
    log.remove()
    log.add(sys.stderr, level="INFO")
    asyncio.run(run())
```



Here's what we got when we ran it against `web-scraping.dev`:

text

```text
2026-11-12 10:55:44.922 +02:00 | INFO     | __main__:scrape:119 - scraping 1 urls
2026-11-12 10:55:46.290 +02:00 | INFO     | __main__:run:137 - depth 0: scraped 1 pages and failed 0
2026-11-12 10:55:46.292 +02:00 | INFO     | __main__:parse:112 - found 9 urls to follow (from total 17)
2026-11-12 10:55:46.292 +02:00 | INFO     | __main__:scrape:119 - scraping 9 urls
2026-11-12 10:55:48.350 +02:00 | INFO     | __main__:run:137 - depth 1: scraped 9 pages and failed 0
2026-11-12 10:55:48.356 +02:00 | INFO     | __main__:parse:112 - found 15 urls to follow (from total 34)
2026-11-12 10:55:48.356 +02:00 | INFO     | __main__:scrape:119 - scraping 15 urls
2026-11-12 10:55:50.666 +02:00 | INFO     | __main__:run:137 - depth 2: scraped 15 pages and failed 0
2026-11-12 10:55:50.680 +02:00 | INFO     | __main__:parse:112 - found 46 urls to follow (from total 74)
2026-11-12 10:55:50.680 +02:00 | INFO     | __main__:scrape:119 - scraping 25 urls
2026-11-12 10:55:54.686 +02:00 | INFO     | __main__:run:137 - depth 3: scraped 25 pages and failed 0
2026-11-12 10:55:54.705 +02:00 | INFO     | __main__:parse:112 - found 44 urls to follow (from total 109)
200 https://web-scraping.dev/
404 https://web-scraping.dev/api/graphql
200 https://web-scraping.dev/products
200 https://web-scraping.dev/mcp-tools
200 https://web-scraping.dev/testimonials
...
```



It took about 11 seconds and stopped at our 50 page limit, with 44 more URLs left to follow. Without the limit it kept going for about 40 seconds and finished on its own after 162 pages.

All 37 URLs from `/sitemap.xml` were in there, products 26 to 28 included. Most of the other 125 were `?variant=` product links and paging or category pages.

It didn't find `/js-links` or `/session-vortex` though, the two pages that only came up in the Wayback Machine earlier.

One thing we fixed from the 2022 version is the extension list. It had `pdf` and `png` without the dot, and here's what `posixpath.splitext()` returns:

python

```python
>>> posixpath.splitext("/assets/pdf/tos.pdf")
('/assets/pdf/tos', '.pdf')
```



So, the old list never matched anything. For more on skipping files, see our guide on how to [ignore non-HTML URLs](https://scrapfly.io/blog/answers/how-to-ignore-non-html-urls-when-web-crawling).

[How to Crawl the Web with PythonIntroduction to web crawling with Python. What is web crawling? How it differs from web scraping? And a deep dive into code, building our own crawler and an example project crawling Shopify-powered websites.](https://scrapfly.io/blog/posts/crawling-with-python)

### Staying on One Domain

The `is_valid_domain()` check is what keeps our crawler on `web-scraping.dev`. With the `DEBUG` logs on, we can see what it drops:

text

```text
2026-11-12 10:56:33.008 +02:00 | DEBUG    | __main__:filter:61 - drop domain missmatch https://github.com/scrapfly
2026-11-12 10:56:33.008 +02:00 | DEBUG    | __main__:filter:64 - drop ignored extension https://web-scraping.dev/sitemap.xml
2026-11-12 10:56:34.784 +02:00 | DEBUG    | __main__:filter:64 - drop ignored extension https://web-scraping.dev/assets/pdf/tos.pdf
```



On the 50 page run that was 18 offsite links, 5 files and 94 duplicates.

A substring check like `"web-scraping.dev" in url` would let `web-scraping.dev.attacker.com` through. `tldextract` splits it up properly:

python

```python
>>> tldextract.extract("https://web-scraping.dev.attacker.com/")
ExtractResult(subdomain='web-scraping.dev', domain='attacker', suffix='com', is_private=False)
```



`www.web-scraping.dev` gets dropped too, as its subdomain is `www` and ours is empty.

The 2022 version of this check used `registered_domain`. On `tldextract` 5.4.0 that prints a warning:

text

```text
DeprecationWarning: The 'registered_domain' property is deprecated and will be removed in the next major version. Use 'top_domain_under_public_suffix' instead, which has the same behavior but a more accurate name.
```



### Normalizing URLs

`canonicalize_url()` from `w3lib` is what our filter uses for deduplication:

python

```python
>>> canonicalize_url("https://web-scraping.dev/products?page=2&category=apparel#reviews")
'https://web-scraping.dev/products?category=apparel&page=2'
>>> canonicalize_url("https://Web-Scraping.dev/products/?utm_source=x")
'https://web-scraping.dev/products/?utm_source=x'
```



It sorts the query, lowercases the domain and drops the `#reviews` fragment. Query parameters like `page` stay, and so do the `utm_` ones and the trailing slash.

On `web-scraping.dev`, `/products/` is a 307 redirect to `/products`.

Scrapy comes with most of this built in. Let's take a look at it next.



## Crawling With Scrapy

Scrapy is a popular web scraping framework in Python and it has a great feature set writing a Python web crawler. Scrapy's web spider class [CrawlSpider](https://docs.scrapy.org/en/latest/topics/spiders.html#crawlspider) implements the same crawling algorithm we covered in the previous section.

Scrapy comes with many batteries-included features like bad response retrying and efficient request scheduling. However, Scrapy, being a full framework, can be difficult to patch and integrate with other python technologies.

For our URL list, all we need is a `CrawlSpider` with one `Rule` that follows every link and yields the URL and status of each page:

python

```python
import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class DomainSpider(CrawlSpider):
    name = "domain_spider"
    allowed_domains = ["web-scraping.dev"]
    start_urls = ["https://web-scraping.dev/"]

    rules = (
        Rule(LinkExtractor(allow_domains=["web-scraping.dev"]), callback="parse_item", follow=True),
    )

    def parse_start_url(self, response):
        return self.parse_item(response)

    def parse_item(self, response):
        yield {
            "url": response.url,
            "status": response.status,
        }

if __name__ == "__main__":
    process = CrawlerProcess(
        settings={
            "ROBOTSTXT_OBEY": True,
            "CLOSESPIDER_PAGECOUNT": 10,
            "DOWNLOAD_DELAY": 1.0,
            "CONCURRENT_REQUESTS": 2,
            "FEEDS": {
                "scrapy_urls.csv": {
                    "format": "csv",
                    "mode": "overwrite",
                }
            },
            "LOG_LEVEL": "INFO",
        }
    )
    process.crawl(DomainSpider)
    process.start()
```



`allowed_domains` is a safety feature that restricts this spider to crawling only particular domains. It reduces accidental errors where the spider could wander off and scrape some other domains accidentally.

There are two ways to run Scrapy spiders, either through the `scrapy` command or by explicitly calling Scrapy via a Python script. We're using a script with `CrawlerProcess` here.

Here's what we got when we checked `ROBOTSTXT_OBEY`:

shell

```shell
$ python -c "from scrapy.settings import default_settings; print(default_settings.ROBOTSTXT_OBEY)"
False
$ scrapy startproject demo > /dev/null && grep ROBOTSTXT_OBEY demo/demo/settings.py
ROBOTSTXT_OBEY = True
```



With it set in our script, the stats at the end of the run have these lines:

text

```text
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
```



When we removed the setting, they were gone and the spider never requested `robots.txt` at all.

The `parse_start_url()` method is there for the homepage. Here's how the CSV started when we left it out:

csv

```csv
url,status
https://web-scraping.dev/sitemap.xml,200
https://web-scraping.dev/cart,200
```



Running it with Scrapy 2.20.0 took about 12 seconds and here is what we got in `scrapy_urls.csv`:

csv

```csv
url,status
https://web-scraping.dev/,200
https://web-scraping.dev/sitemap.xml,200
https://web-scraping.dev/cart,200
https://web-scraping.dev/login,200
https://web-scraping.dev/mcp-tools,200
https://web-scraping.dev/file-download,200
https://web-scraping.dev/credentials,200
https://web-scraping.dev/testimonials,200
https://web-scraping.dev/reviews,200
https://web-scraping.dev/products,200
```



Scrapy provides brilliant logs that log everything that the Scrapy engine is performing, as well as logging the returned results.

Moreover, Scrapy attaches some useful scrape statistics, such as how many items were scraped, how long it took for our scraper to finish, etc. Here are a few from our run:

text

```text
 'dupefilter/filtered': 90,
 'finish_reason': 'closespider_pagecount',
 'item_scraped_count': 10,
```



We also got `/sitemap.xml` in the list, so we checked the extensions `LinkExtractor` skips:

python

```python
>>> from scrapy.linkextractors import IGNORED_EXTENSIONS
>>> "pdf" in IGNORED_EXTENSIONS, "xml" in IGNORED_EXTENSIONS
(True, False)
```



Scrapy also has a [SitemapSpider](https://docs.scrapy.org/en/latest/topics/spiders.html#sitemapspider) for crawling straight from sitemaps.

[Web Scraping With Scrapy: The Complete Guide in 2026Build and run a Scrapy 2.18 project with async start(), pagination, selector tests, item validation, pipelines, and JSON Lines export.](https://scrapfly.io/blog/posts/web-scraping-with-scrapy)

However, using an HTTP client we may not be able to scrape highly dynamic content like that of javascript-powered web apps or single-page applications (SPAs). Let's take a look at what else a link crawl can miss next



## Common Crawling Problems

Web crawling is simply more resource-intensive and harder to develop as we need to consider a whole series of new problems related to this exploration component.

When crawling or broad-crawling we have significantly less control over our program's flow than when web scraping. Some links might be protected against web scrapers and some might be dynamic javascript-powered web pages.

`web-scraping.dev` has a test page for most of these problems.

Our crawler never reached `/js-links` or `/session-vortex`. We only know about them from the Wayback Machine.

When we requested `/js-links` directly with `httpx`, our XPath didn't find a single link:

python

```python
>>> response = httpx.get("https://web-scraping.dev/js-links")
>>> Selector(text=response.text).xpath("//a/@href").getall()
[]
```



Its only link, to `/js-links-target`, is added by a `<script>` after the page loads.

To crawl dynamic javascript websites and SPAs crawler needs to be able to execute javascript. The easiest approach to this is to use a headless browser automated through [Web Scraping With Playwright in 2026: A Python Guide](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python#what-is-playwright), [Web Scraping with Selenium and Python](https://scrapfly.io/blog/posts/web-scraping-with-selenium-and-python#what-is-selenium) or [How to Web Scrape with Puppeteer and NodeJS in 2026](https://scrapfly.io/blog/posts/web-scraping-with-puppeteer-and-nodejs#puppeteer-overview).

For more, see [why a scraper sees different content than your browser](https://scrapfly.io/blog/answers/why-cant-scraper-see-content).

Modern websites use hidden APIs to render dynamic data on the fly through background requests. Often this is used in product pagination, search functionality and other dynamic page parts.

On `/testimonials`, the second page comes from an `hx-get` attribute and not an `<a>` link:

shell

```shell
$ curl -s https://web-scraping.dev/testimonials | grep -o -E 'hx-(get|trigger)="[^"]*"'
hx-get="https://web-scraping.dev/api/testimonials?page=2"
hx-trigger="revealed"
```



That API returned a 422 for us until we added a `Referer` header. See our guide on [scraping hidden APIs](https://scrapfly.io/blog/posts/how-to-scrape-hidden-apis) for more.

When web scraping, we're often limited by the website's technical capabilities. We can't scrape too fast without being blocked or overwhelming smaller websites.

Here's what we got from 40 requests in a row to `/rate-limited`:

shell

```shell
$ for i in $(seq 1 40); do curl -s -o /dev/null -w '%{http_code}\n' https://web-scraping.dev/rate-limited; done | sort | uniq -c
     29 200
     11 429
```



`/blocked` is trickier. It returns a 200 status code with a block page:

python

```python
>>> response = httpx.get("https://web-scraping.dev/blocked")
>>> response.status_code, Selector(text=response.text).css("title::text").get()
(200, "web-scraping.dev - You've been blocked")
```



Our full crawl from earlier saved it as a normal `200` page, and `/blocked?persist` too. Any links on a blocked page are lost as well.

[How to Bypass Anti-Bot Protection When Web ScrapingLearn how anti-bot systems detect scrapers and 5 universal bypass techniques including proxy rotation, fingerprinting, and fortified headless browsers.](https://scrapfly.io/blog/posts/how-to-bypass-anti-bot-protection-when-web-scraping)

Some pages are left out on purpose. The `robots.txt` file of this website disallows `/robots-disallowed` for everyone and `/api/` for Googlebot and Bingbot:

python

```python
>>> rp.can_fetch("*", "https://web-scraping.dev/robots-disallowed")
False
>>> rp.can_fetch("Googlebot", "https://web-scraping.dev/api/graphql")
False
>>> rp.can_fetch("*", "https://web-scraping.dev/api/graphql")
True
```



Then there are pages that never run out of links. The calendar pages have a prev and next link, and here's how far they went for us:

shell

```shell
$ for m in 2000/1 2024/1 2050/1 9999/12; do curl -s -o /dev/null -w "$m %{http_code}\n" https://web-scraping.dev/calendar/$m; done
2000/1 200
2024/1 200
2050/1 200
9999/12 404
```



`/session-vortex` is the other one. Here are a few of its links:

shell

```shell
$ curl -s https://web-scraping.dev/session-vortex | grep -o 'session-vortex?sid=[^"]*' | head -3
session-vortex?sid=fdaed544-6650-42c3-b076-d0d0090ad072
session-vortex?sid=7ad53068-fba4-4857-a627-62812379a0fe
session-vortex?sid=6c75cfac-2d08-4d7a-a580-729d2792bcc7
```



Two loads in a row had none in common, and `canonicalize_url()` keeps the `sid` parameter. `url_query_cleaner()` from `w3lib` can drop it:

python

```python
>>> url_query_cleaner("https://web-scraping.dev/session-vortex?sid=0a9a3ecb-3c3e-4a03-ad65-b252d25bf312", ["sid"], remove=True)
'https://web-scraping.dev/session-vortex'
```



Pages a crawl can't reach might still be in search engines or the Wayback Machine. Let's take a look at those next.



Scrapfly

#### Scale your web scraping effortlessly

Scrapfly handles proxies, browsers, and anti-bot bypass — so you can focus on data.

[Try Free →](https://scrapfly.io/register)

## Search Engines and the Wayback Machine

Search engines keep their own index of every website they crawl, and a `site:web-scraping.dev` search on Google lists what it has for this one. Adding a path like `site:web-scraping.dev/product/` narrows it down.

Google's own [site: operator docs](https://developers.google.com/search/docs/monitor-debug/search-operators/all-search-site) say it "doesn't necessarily return all the URLs that are indexed under the prefix specified in the query".

Without a search term the results aren't ranked either. Google shows the shortest URL first, "but otherwise the results are relatively random".

The Wayback Machine is easier to work with from Python as it has a CDX API that lists every archived URL. Here's a script that pulls them for `web-scraping.dev` and checks what each one returns today:

python

```python
from collections import Counter
import httpx

response = httpx.get(
    "https://web.archive.org/cdx/search/cdx",
    params={"url": "web-scraping.dev/*", "fl": "original", "collapse": "urlkey"},
    timeout=60,
)
urls = response.text.splitlines()
print(f"found {len(urls)} archived urls")

# skip images, css and the /sitemap-page/ test urls
pages = [url for url in urls if "/assets/" not in url and "/sitemap-page/" not in url]
print(f"checking {len(pages)} pages")

statuses = Counter()
with httpx.Client(follow_redirects=True, timeout=20) as client:
    for url in pages:
        response = client.get(url)
        statuses[response.status_code] += 1
print(statuses)
```



Running it took close to 2 minutes and here is what we got:

text

```text
found 2632 archived urls
checking 275 pages
Counter({200: 218, 404: 38, 422: 17, 405: 2})
```



The 422s were `/api/reviews` and `/api/testimonials` calls, and the two 405s were `/api/login` and `/api/download-file`. The 404s include old `/banners/*.png` images and broken captures like `/products&quot;`.



## Merging Everything Into One List

We can successfully collect URLs from the sitemaps, our crawlers and the Wayback Machine. Next, let's combine them into one list.

We'll use `canonicalize_url()` again, with the sitemap and the `scrapy_urls.csv` file from the Scrapy spider:

python

```python
import csv
import httpx
from parsel import Selector
from w3lib.url import canonicalize_url

response = httpx.get("https://web-scraping.dev/sitemap.xml")
selector = Selector(response.text)
selector.remove_namespaces()
sitemap_urls = {canonicalize_url(url) for url in selector.xpath("//url/loc/text()").getall()}

with open("scrapy_urls.csv") as f:
    crawl_urls = {canonicalize_url(row["url"]) for row in csv.DictReader(f)}

print(f"total: {len(sitemap_urls | crawl_urls)}")
print(f"in both: {len(sitemap_urls & crawl_urls)}")
print(f"sitemap only: {len(sitemap_urls - crawl_urls)}")
print(f"crawl only: {sorted(crawl_urls - sitemap_urls)}")
```



Here is what we got:

text

```text
total: 39
in both: 8
sitemap only: 29
crawl only: ['https://web-scraping.dev/mcp-tools', 'https://web-scraping.dev/sitemap.xml']
```



The 29 sitemap-only URLs are all 28 products plus `/blocked`. Our Scrapy spider stopped after 10 pages.

### Finding Orphan Pages

Orphan pages are pages that no other page links to. Going by the 10-page crawl, all 28 products would look like orphans.

Here's the same merge with the 162 pages from our full `httpx` crawl:

text

```text
total: 162
in both: 37
sitemap only: 0
crawl only: ['https://web-scraping.dev/api/graphql', 'https://web-scraping.dev/blocked?persist=', ...
```



Though with crawling and filtering it's possible to discover all of the pages.

Another approach is to discover pages through their IDs. Each product on `web-scraping.dev` is assigned an incremental ID, and using HEAD-type requests we can easily poke each of these IDs to see whether they lead to product pages.

We tried that, but `web-scraping.dev` answered every HEAD request with a `405 Method Not Allowed`. With plain GET requests, `/product/28` was the last product page and `/product/29` came back as a 404.

If all that seems like too much trouble, let Scrapfly handle it for you! Let's take a look at Scrapfly's Crawler API next.



## Crawling With Scrapfly's Crawler API



We can easily power up our crawlers by replacing all of the code above with Scrapfly's [Crawler API](https://scrapfly.io/products/crawler-api). It runs the whole crawl on Scrapfly's side and can be used through the [scrapfly-sdk](https://scrapfly.io/docs/sdk/python) which can be installed through pip:

shell

```shell
$ pip install scrapfly-sdk
```



The config takes the same kind of limits we set on our own crawlers. Here's our `web-scraping.dev` crawl with a 50 page limit:

python

```python
from scrapfly import ScrapflyClient, CrawlerConfig, Crawl

client = ScrapflyClient(key="YOUR_SCRAPFLY_KEY")
config = CrawlerConfig(
    url="https://web-scraping.dev/",
    page_limit=50,
    max_depth=3,
    use_sitemaps=True,  # read the sitemaps too
    rendering_delay=0,  # plain HTTP requests, no browser
    ignore_base_path_restriction=True,
    follow_internal_subdomains=False,
)

crawl = Crawl(client, config).crawl().wait()
state = crawl.status().state
print(state, state.stop_reason)
for page in crawl.warc().iter_responses():
    print(page.status_code, page.url)
```



The `state` has counters like `urls_visited`, `urls_failed` and `urls_skipped`. Its `stop_reason` is `no_more_urls` when the crawl ran out of links, `page_limit` when it hit our limit and `max_duration` when it ran out of time.

Some defaults are different from our own crawler. The Crawler API stays under the start URL's path unless `ignore_base_path_restriction` is on, and it follows subdomains like `blog.web-scraping.dev` unless `follow_internal_subdomains` is off.

It also doesn't read sitemaps unless `use_sitemaps` is on. For websites with anti-bot protection, there's an `unblocker` option as well.

Once it's done, `crawl.urls(status="failed")` lists the URLs that failed, and the WARC file has every page it downloaded with its status code. For more, see the [Crawler API docs](https://scrapfly.io/docs/crawler-api/getting-started).

To wrap up this article, let's take a look at some frequently asked questions about finding all URLs on a domain.



## FAQ

Do subdomains count when you list all URLs on a domain?

Depends on the crawler. Subdomains like `blog.web-scraping.dev` are often entirely different websites so our Python crawler treats them as offsite links. Scrapy's `allowed_domains` and Scrapfly's Crawler API are the other way around and follow subdomains unless they're excluded.







How can I find all pages on a website for free without code?

The easiest way is to check the website's sitemaps. They can usually be found at `/sitemap.xml` or in the `/robots.txt` file and any web browser can open them. Alternatively, public search indexes like google.com or bing.com can be used with queries like `site:web-scraping.dev`. Finally, the Wayback Machine can list all archived URLs of a website at `web.archive.org/web/*/web-scraping.dev/*`.







How do I find a hidden page on a website I don't own?

Finding hidden pages is a bit more difficult since nothing on the website links to them. The Wayback Machine is the best bet as it often keeps records of old and unlinked pages, that's how we found `/session-vortex` on `web-scraping.dev`. Another approach is to discover pages through their IDs. Many websites assign incremental IDs to pages like products so we can poke each ID and see which ones lead to real pages.







How many URLs can one XML sitemap list?

Sitemaps are usually XML documents that list URLs by category in batches of 50 000. The [sitemap protocol](https://www.sitemaps.org/protocol.html) limits each file to 50 000 URLs or 50MB uncompressed so bigger websites use sitemap hubs (indexes) that link to multiple sitemap files.









## Summary

In this article, we've taken an in-depth look at how to find all URLs on a domain using Python. We started with the sitemap system, which we found through the `/robots.txt` file and parsed using `httpx` and `parsel`.

Sitemaps are a quick way to get an overview of the website structure. Though, not every website supports them and they only list what the website owner put in them.

Then, we built a web crawler with Python alongside with a few Python packages and took a look at Scrapy's `CrawlSpider` as well. To solidify our knowledge we crawled `web-scraping.dev` with both and compared the results with its sitemaps.

We also explained common crawling problems like javascript-powered pages, blocking and crawler traps, and how search engines and the Wayback Machine can help with pages a crawler can't reach.

Finally, we've merged the sitemap and crawl results into one list. For bigger websites, Scrapfly's Crawler API can run the whole crawl for you, so give it a shot!



Legal Disclaimer and Precautions

This tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect:

- Do not scrape at rates that could damage the website.
- Do not scrape data that's not available publicly.
- Do not store PII of EU citizens protected by GDPR.
- Do not repurpose *entire* public datasets which can be illegal in some countries.

Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer.

 

   [  Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents















 

  Table of Contents

- [Key Takeaways](#key-takeaways)
- [Where to Find All URLs of a Domain?](#where-to-find-all-urls-of-a-domain)
- [Where to Find Sitemaps?](#where-to-find-sitemaps)
- [Crawling a Domain With Python](#crawling-a-domain-with-python)
- [Staying on One Domain](#staying-on-one-domain)
- [Normalizing URLs](#normalizing-urls)
- [Crawling With Scrapy](#crawling-with-scrapy)
- [Common Crawling Problems](#common-crawling-problems)
- [Search Engines and the Wayback Machine](#search-engines-and-the-wayback-machine)
- [Merging Everything Into One List](#merging-everything-into-one-list)
- [Finding Orphan Pages](#finding-orphan-pages)
- [Crawling With Scrapfly's Crawler API](#crawling-with-scrapfly-s-crawler-api)
- [FAQ](#faq)
- [Summary](#summary)
 
    Join the Newsletter  Get monthly web scraping insights 

 

  



Scale Your Web Scraping

Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits.

  No credit card required  1,000 free API credits  Anti-bot bypass included 

 [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) 

 Not ready? [Get our newsletter](#newsletter-modal) instead. 

 

 

## Related Articles

 [     

 python data-parsing 

### How to Parse XML

In this article, we'll explain about XML parsing. We'll start by defining XML files, their format and how to navigate th...

 

 ](https://scrapfly.io/blog/posts/how-to-parse-xml) [     

 python scrapeguide 

### How to Scrape Domain.com.au Real Estate Property Data

We'll be taking a look at another real estate target in Australia - domain.com.au. To scrape real estate data we'll be u...

 

 ](https://scrapfly.io/blog/posts/how-to-scrape-domain-com-au-real-estate-property-data) [     

 data-parsing 

### JSON vs XML: Key Differences and Modern Uses

JSON and XML are two major data formats encountered in web development — here's how they differ and which is one better ...

 

 ](https://scrapfly.io/blog/posts/json-vs-xml) 

  

## Related Questions

- [ Q How to Use cURL Config Files? ](https://scrapfly.io/blog/answers/how-to-set-curl-config-file)
- [ Q Scraper doesn't see the data I see in the browser - why? ](https://scrapfly.io/blog/answers/why-cant-scraper-see-content)
 
  



   



 Scale your web scraping effortlessly, **1,000 free credits** [Start Free](https://scrapfly.io/register)