Handling Crawl-delay and Sitemap: Directives in robots.txt #

The two most useful lines in many robots.txt files — Crawl-delay and Sitemap: — are the two that Python’s standard urllib.robotparser handles least. This page shows how to extract both reliably by reading raw directive lines, apply Crawl-delay to your request pacing, and seed a crawl frontier from the sitemaps a site advertises. It sits under parsing robots.txt programmatically within the Compliance & Ethical Crawling Foundations section, and it picks up where basic allow/deny parsing leaves off: turning the non-Allow/Disallow directives into concrete pacing and scheduling behaviour.

Problem Framing #

Crawl-delay and Sitemap: are extensions, not part of the original robots exclusion standard, and tooling treats them inconsistently. urllib.robotparser added a crawl_delay() accessor in Python 3.6 and a site_maps() accessor, but both are brittle: crawl_delay() returns None for the wildcard * group in several real files it fails to associate correctly, and site_maps() returns None rather than an empty list when no sitemap is present, which trips up naive callers. Neither gives you the raw line if parsing quietly drops it.

In production this surfaces as two silent failures. First, a site politely asks for a ten-second gap between requests and your crawler ignores it because the parser handed back None, so you hammer the endpoint and earn a block. Second, a site publishes a comprehensive sitemap index that would let you discover every URL cheaply, and you never read it because your parser only exposed Disallow rules. Both are avoidable by reading the file’s raw lines alongside the standard parser.

Step-by-Step Implementation #

1. Fetch robots.txt once, keep the raw text. You need the bytes both for the standard parser and for line-level extraction of the non-standard directives.
2. Feed the text to urllib.robotparser for authoritative can_fetch() allow/deny decisions — do not reimplement that logic.
3. Scan raw lines yourself for Crawl-delay and Sitemap, because these are the fields the standard parser handles unreliably. Match case-insensitively and split on the first colon.
4. Resolve the applicable Crawl-delay by tracking which User-agent group each directive belongs to, preferring a group matching your agent token over the * fallback.
5. Collect every Sitemap: URL — these are global to the file, not scoped to a user-agent group — and hand them to your frontier seeder.

import urllib.robotparser
import urllib.request

def parse_robots(robots_url: str, my_agent: str):
    raw = urllib.request.urlopen(robots_url, timeout=10).read().decode("utf-8", "replace")

    # Standard parser for authoritative allow/deny decisions.
    rp = urllib.robotparser.RobotFileParser()
    rp.parse(raw.splitlines())

    sitemaps, crawl_delays, current_agents = [], {}, ["*"]
    for line in raw.splitlines():
        line = line.split("#", 1)[0].strip()      # strip comments
        if not line or ":" not in line:
            continue
        field, _, value = line.partition(":")
        field, value = field.strip().lower(), value.strip()
        if field == "user-agent":
            current_agents = [value.lower()]
        elif field == "crawl-delay":
            for agent in current_agents:
                crawl_delays[agent] = float(value)   # seconds between requests
        elif field == "sitemap":
            sitemaps.append(value)                   # absolute URL, file-global

    # Prefer a delay targeting our token; fall back to the wildcard group.
    delay = crawl_delays.get(my_agent.lower(), crawl_delays.get("*"))
    return rp, delay, sitemaps

Applying Crawl-delay to pacing #

A Crawl-delay of N means “leave at least N seconds between the start of consecutive requests to this host.” Feed it into your limiter as a per-host minimum interval rather than a blind sleep, so it composes cleanly with the rest of your polite rate limiting:

import time

class HostPacer:
    def __init__(self, crawl_delay: float | None, floor: float = 1.0):
        # Honour the site's delay, but never crawl faster than our own floor.
        self.interval = max(crawl_delay or 0.0, floor)
        self._last = 0.0

    def wait(self):
        gap = time.monotonic() - self._last
        if gap < self.interval:
            time.sleep(self.interval - gap)
        self._last = time.monotonic()

Seeding the frontier from sitemaps #

Sitemap URLs may point at a <sitemapindex> (a list of child sitemaps) or a <urlset> (actual page URLs). Resolve the index, then enqueue the leaf URLs into your scheduler — the same frontier you would manage with distributed crawl scheduling and queues:

import gzip, requests
from lxml import etree

def iter_sitemap_urls(sitemap_url: str):
    body = requests.get(sitemap_url, timeout=15).content
    if sitemap_url.endswith(".gz"):
        body = gzip.decompress(body)
    root = etree.fromstring(body)
    ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
    if root.tag.endswith("sitemapindex"):
        for loc in root.findall(".//sm:sitemap/sm:loc", ns):
            yield from iter_sitemap_urls(loc.text.strip())   # recurse into children
    else:
        for loc in root.findall(".//sm:url/sm:loc", ns):
            yield loc.text.strip()

Verification & Testing #

Point the parser at a fixture and assert both directives resolve. A minimal robots.txt served from a local file or httpbin-style echo makes this deterministic:

FIXTURE = """User-agent: *
Crawl-delay: 10
Disallow: /private

Sitemap: https://example.com/sitemap_index.xml
"""

def test_extracts_delay_and_sitemap(tmp_path):
    rp = urllib.robotparser.RobotFileParser()
    rp.parse(FIXTURE.splitlines())
    # Re-run the raw scan from parse_robots against FIXTURE...
    assert delay == 10.0
    assert "https://example.com/sitemap_index.xml" in sitemaps
    assert rp.can_fetch("*", "https://example.com/public") is True

To confirm pacing behaves, log timestamps across ten HostPacer.wait() calls and assert consecutive gaps are >= interval. To confirm sitemap seeding, count the URLs yielded from a known sitemap and compare against the <loc> count in the raw XML.

Compliance & Operational Guardrails #

• Crawl-delay is a request, but honour it as a floor. Treating it as advisory and ignoring it is precisely the behaviour that gets crawlers characterised as abusive; enforce the larger of the site’s value and your own polite baseline.
• A published sitemap is an invitation, not blanket consent. It tells you which URLs exist, but Disallow rules and Terms of Service still govern which you may fetch — always gate sitemap URLs through can_fetch().
• Cap absurd delays. A malformed or hostile Crawl-delay: 86400 should trip a review rather than silently stalling your pipeline for a day; clamp to a sane ceiling and alert.

Crawl-Delay Values That Need Clamping #

Published delay values are not always reasonable, and a parser that applies them literally will occasionally stall a crawl for hours. Three cases recur often enough to handle explicitly.

A very large value — Crawl-delay: 3600, or the occasional 86400 — is usually a site expressing “we would rather you did not crawl this at all” in the only vocabulary the file offers. Applying it literally means one request per hour, which for a catalogue of ten thousand URLs is a crawl measured in years. The correct handling is to clamp the applied delay to a configured ceiling and to flag the host for review, because the honest reading of that value is closer to a refusal than to a pacing hint.

A fractional value — Crawl-delay: 0.5 — is well-formed and should be honoured, but only as a floor. Never round it down to zero, and never treat a fractional value as a licence to ignore your own configured minimum.

A malformed value — Crawl-delay: 10 seconds, an empty value, or a comma decimal separator — must not silently become zero. Parse defensively and fall back to the configured floor rather than to no delay at all, which is the direction a naive float() in a try block usually fails.

DELAY_CEILING = 60.0        # applied maximum; above this, review the host
DELAY_FLOOR = 1.0           # our own politeness minimum

def resolve_crawl_delay(raw: str | None) -> tuple[float, str]:
    """Return (applied delay seconds, reason). Never returns less than the floor."""
    if raw is None:
        return DELAY_FLOOR, "no directive; using configured floor"
    try:
        value = float(str(raw).strip().replace(",", "."))
    except ValueError:
        return DELAY_FLOOR, f"unparseable value {raw!r}; using configured floor"
    if value <= 0:
        return DELAY_FLOOR, "non-positive value; using configured floor"
    if value > DELAY_CEILING:
        return DELAY_CEILING, f"declared {value}s exceeds ceiling; clamped and flagged for review"
    return max(value, DELAY_FLOOR), f"honouring declared {value}s"

Returning the reason alongside the value is what makes this auditable. The applied delay ends up in the crawl log next to the sentence explaining how it was derived, so a later question — “why were we fetching this host every second when its rules said ten?” — has an answer that does not require re-reading the code.

Sitemaps Are Global, Not Group-Scoped #

A frequent parsing error is attaching Sitemap: entries to whichever User-agent group they happen to appear under. The directive is file-scoped: every sitemap declared anywhere in the document applies regardless of which group your crawler matched. Collect them during a full pass over the file rather than inside the group selection routine.

Treat a declared sitemap as a discovery source, not an authorisation. URLs found in it are still subject to the same Disallow rules, the same terms of service mapping, and the same per-host budget as any other URL — a site listing a path in its sitemap while disallowing it in the same file has published a contradiction, and the restrictive rule wins. Feed discovered URLs through the normal frontier so the politeness-aware scheduler applies the resolved delay to them exactly as it would to a link found by crawling.

Common Mistakes #

1. Trusting crawl_delay() alone. It returns None for cases where the directive is plainly present in the file; always cross-check with a raw-line scan before assuming a site set no delay.
2. Scoping Sitemap: to a user-agent group. Sitemap directives are file-global; associating them with the nearest User-agent block drops sitemaps declared before any group.
3. Enqueuing sitemap URLs without deduplication. Sitemap indexes overlap and repeat; feed the URLs through your seen-set before they hit the frontier or you will re-crawl thousands of duplicates.

Frequently Asked Questions #

Does Google honour Crawl-delay? #

No — Googlebot explicitly ignores Crawl-delay and manages its own rate via Search Console instead. That does not make the directive safe to ignore for your own crawler: many smaller sites rely on it as their only throttling signal, and disregarding it is exactly the conduct that provokes blocks and complaints.

Where should I look for sitemaps if robots.txt has no Sitemap: line? #

Try the conventional /sitemap.xml at the site root, then check for a <link rel="sitemap"> in page markup. Absence of any advertised sitemap is not permission to crawl more aggressively — fall back to polite link discovery within the allowed paths.

How do I handle a gzipped or nested sitemap index? #

Decompress .gz payloads before parsing, and recurse when the root element is <sitemapindex> rather than <urlset>, as shown above. Guard the recursion with a depth limit and your seen-set so a malformed index that references itself cannot loop forever.

Related guides #

• Parsing robots.txt Programmatically — the parent technique covering the full exclusion standard.
• How to Parse robots.txt With Python urllib — the allow/deny foundation this page builds on.
• Distributed Crawl Scheduling and Queues — where sitemap-seeded URLs enter your frontier at scale.