Detecting and Respecting noindex / nofollow Directives #

A publisher can allow a crawler to fetch a page while still signalling that the page should not be indexed, or that its outbound links should not be followed. Those signals — the noindex and nofollow robots directives — live in the HTML <meta> tag, in the X-Robots-Tag HTTP header, and in the rel attribute of individual links. Honouring them is a narrow but load-bearing part of running a defensible crawler, and it sits under the Mapping Terms of Service for Scrapers technique within the broader Compliance & Ethical Crawling Foundations section. This page covers how to detect all three carriers and wire them into link-following logic.

Problem Framing #

robots.txt controls access — whether you may request a URL at all. The noindex/nofollow family is different: the server has already served you the page, and is now expressing intent about what you do with it. noindex says “do not add this page to your index / dataset”; nofollow says “do not use the links on this page (or this specific link) to discover more URLs, and do not pass ranking signal through them.”

A header directive binds before the body is even parsed.

In production this arises constantly. A site marks its faceted-search results noindex to keep them out of search engines but still wants humans to browse them. A forum adds rel="nofollow" to user-submitted links to disavow spam. A paginated archive sets X-Robots-Tag: noindex, nofollow at the CDN layer so it never appears in the header of the served HTML you might be caching. A crawler that ignores these is not just impolite — it is disregarding an explicit, machine-readable expression of the publisher’s wishes, which weakens any argument that your collection was authorised. Because the header form and the meta form can disagree, and because directives can be scoped to a named bot, detection has to check every carrier before a link is queued.

Step-by-Step Implementation #

Read the X-Robots-Tag response header first. It arrives before you parse the body and can apply noindex/nofollow to non-HTML resources (PDFs, images) that have no place for a meta tag. A header may carry multiple comma-separated tokens and may be bot-scoped as googlebot: noindex.
Parse the <meta name="robots"> tag (and any bot-specific <meta name="yourbot">) from the <head>. Treat the wildcard robots name as applying to everyone.
Merge header and meta directives — the union of both applies. If either says noindex, the page is not stored; if either says nofollow, no link on the page is enqueued.
When the page is followable, inspect each link’s rel attribute for nofollow (and related sponsored / ugc values) and skip those individual targets.
Record the decision against the URL so audits can show why a page was dropped or a link was not followed.

Discard happens before storage, so noindex content never lands in the sink.

import re
from dataclasses import dataclass
from bs4 import BeautifulSoup

MY_BOT = "compliantbot"

@dataclass
class RobotsDirectives:
    noindex: bool = False
    nofollow: bool = False

def _tokens_for_bot(field_value: str) -> str:
    """A directive line may be 'noindex, nofollow' or 'googlebot: noindex'."""
    if ":" in field_value:
        bot, _, directives = field_value.partition(":")
        if bot.strip().lower() not in ("robots", MY_BOT):
            return ""  # scoped to a different bot — ignore
        return directives
    return field_value

def parse_directives(headers: dict, html: str) -> RobotsDirectives:
    d = RobotsDirectives()

    # 1. X-Robots-Tag header (there may be several, comma-joined by the client)
    for raw in headers.get("X-Robots-Tag", "").split(","):
        toks = _tokens_for_bot(raw).lower()
        d.noindex |= "noindex" in toks or "none" in toks
        d.nofollow |= "nofollow" in toks or "none" in toks

    # 2. <meta name="robots"> / <meta name="compliantbot">
    soup = BeautifulSoup(html, "lxml")
    for tag in soup.find_all("meta"):
        name = (tag.get("name") or "").lower()
        if name in ("robots", MY_BOT):
            content = (tag.get("content") or "").lower()
            d.noindex |= "noindex" in content or "none" in content
            d.nofollow |= "nofollow" in content or "none" in content
    return d

def followable_links(html: str, page_nofollow: bool) -> list[str]:
    """Yield hrefs unless the page or the individual link is nofollow."""
    if page_nofollow:
        return []  # page-level nofollow suppresses every outbound link
    soup = BeautifulSoup(html, "lxml")
    out = []
    for a in soup.find_all("a", href=True):
        rel = {r.lower() for r in (a.get("rel") or [])}
        if rel & {"nofollow", "sponsored", "ugc"}:
            continue  # link-level disavowal — do not enqueue
        out.append(a["href"])
    return out

The none token is shorthand for noindex, nofollow; treating it as both is required for correctness. Note that noindex governs whether you retain the page in your dataset, while nofollow governs whether you expand its links — they are independent and a page can carry either, both, or neither.

Verification & Testing #

Exercise both carriers and the scoped-bot case. httpbin.org/response-headers lets you mint an arbitrary X-Robots-Tag for an integration check.

Scope errors here quietly widen a crawl far beyond what was permitted.

# Confirm the header is read: this should be classified noindex+nofollow.
curl -s -D - "https://httpbin.org/response-headers?X-Robots-Tag=noindex,%20nofollow" -o /dev/null | grep -i x-robots-tag

def test_meta_nofollow_suppresses_links():
    html = '<html><head><meta name="robots" content="nofollow"></head>' \
           '<body><a href="/a">a</a></body></html>'
    d = parse_directives({}, html)
    assert d.nofollow and not d.noindex
    assert followable_links(html, d.nofollow) == []

def test_header_beats_missing_meta():
    d = parse_directives({"X-Robots-Tag": "noindex"}, "<html></html>")
    assert d.noindex and not d.nofollow

def test_rel_nofollow_skips_single_link():
    html = '<a href="/keep">k</a><a href="/spam" rel="nofollow">s</a>'
    assert followable_links(html, page_nofollow=False) == ["/keep"]

def test_directive_scoped_to_other_bot_ignored():
    d = parse_directives({"X-Robots-Tag": "googlebot: noindex"}, "<html></html>")
    assert not d.noindex  # not addressed to us

Compliance & Operational Guardrails #

Directives are intent, not access control. Fetching a noindex page is not a breach by itself, but retaining or republishing it after being told not to index undercuts any claim that your collection respected the publisher’s expressed wishes.
The union rule fails safe. When the header and meta tag disagree, apply the stricter of the two. Never let a body-level index override a header-level noindex.
Respect link-level nofollow at enqueue time, not after the fetch. Discovering and requesting a nofollow target and only then discarding it wastes the target’s bandwidth and defeats the purpose of the signal.
Log every drop. A record of “page X: noindex via X-Robots-Tag, not stored” is exactly the evidence you want in an audit; feed it into your documenting scraping authorization and data lineage trail.

Parsing the Directive Set Correctly #

Both the header and the meta tag carry a comma-separated list of tokens, and both may be scoped to a named crawler. The parsing rules that matter in practice:

Token names are case-insensitive, and whitespace around them is insignificant. NoIndex , NOFOLLOW is the same instruction as noindex,nofollow.
none is shorthand for noindex, nofollow, and all is shorthand for neither. A parser that only looks for the literal strings noindex and nofollow will miss the shorthand entirely.
A named directive overrides the unnamed one for that crawler. X-Robots-Tag: noindex followed by X-Robots-Tag: examplecrawler: all means your crawler is not restricted, provided the name matches your product token.
Multiple headers accumulate. A response may carry several X-Robots-Tag headers; all of them apply, and a client that reads only the first will act on a partial instruction.

def parse_robots_directives(header_values: list[str], agent: str) -> set[str]:
    """Resolve X-Robots-Tag headers into the effective token set for one crawler."""
    generic: set[str] = set()
    specific: set[str] = set()
    token = agent.lower()

    for raw in header_values:
        name, _, rest = raw.partition(":")
        if rest and " " not in name.strip():          # "agentname: tokens"
            target, tokens = name.strip().lower(), rest
        else:
            target, tokens = None, raw
        parsed = {t.strip().lower() for t in tokens.split(",") if t.strip()}
        if "none" in parsed:
            parsed |= {"noindex", "nofollow"}
        (specific if target and target in token else generic).update(parsed)

    effective = specific or generic
    return effective - {"all", "none"}

Returning a set rather than two booleans keeps the parser honest about tokens it does not act on — noarchive, nosnippet, noimageindex, unavailable_after — which can then be logged and reviewed rather than silently discarded. unavailable_after in particular is a retention instruction as much as an indexing one, and a pipeline that stores content past that date is ignoring an explicit expiry the publisher set.

Applying the Result at Two Different Layers #

noindex and nofollow act on different pipeline stages, and conflating them causes both over- and under-collection. noindex governs retention: the document may be fetched, but it must not be stored or served downstream, and any copy already held should be scheduled for removal. nofollow governs discovery: the links on the page must not enter the frontier, but the page’s own content is unaffected.

Implement them where they belong. The retention decision happens between parsing and the sink, so the record never reaches storage; the discovery decision happens in the link-extraction step, before URLs are handed to the crawl frontier. Applying either one late — filtering at export time, or pruning the queue after dispatch — means the pipeline has already done the thing the directive asked it not to do, and the audit record will show it.

Common Mistakes #

Checking only the meta tag. CDNs and app servers frequently set X-Robots-Tag on responses whose HTML carries no robots meta at all — and on non-HTML files that cannot carry one. Header-only noindex is common.
Treating page-level nofollow and link-level rel="nofollow" as the same switch. A page can be fully followable while individual spammy links are disavowed; conversely a page nofollow suppresses every link regardless of rel.
Ignoring bot-scoped directives. X-Robots-Tag: googlebot: noindex addresses Googlebot, not you — but robots: noindex and a directive naming your own agent both do. Parse the optional bot: prefix instead of matching substrings blindly.

Frequently Asked Questions #

Is ignoring a noindex directive illegal? #

Not inherently — noindex is an indexing preference, not an access-control mechanism, so fetching the page is not unauthorised access. The risk is downstream: if a publisher told you not to index a page and you retained and republished it anyway, you have discarded a clear, machine-readable signal of their wishes, which weakens your compliance and ToS position and can support a misuse claim.

Which wins when the X-Robots-Tag header and the meta tag disagree? #

Apply the union and take the stricter outcome. If the header says noindex and the meta tag omits it, the page is still noindex. Search engines resolve conflicts toward the most restrictive directive, and a crawler that wants to demonstrate good faith should do the same rather than cherry-picking the permissive source.

Does nofollow stop me from crawling a URL entirely? #

No. nofollow is a discovery and endorsement signal: it tells you not to use this link to reach a target or to pass ranking value. If you reach the same URL through another followable path, or it is allowed by the target’s own robots.txt, you may still crawl it. nofollow constrains the edge in the link graph, not the destination node.

Mapping Terms of Service for Scrapers — the parent technique this directive handling belongs to.
Parsing robots.txt Programmatically — the access-control layer that runs before any page is fetched.
Documenting Scraping Authorization and Data Lineage — recording each noindex/nofollow decision for audit defensibility.