Compliant Web Scraping

Documenting Scraping Authorization and Data Lineage #

When a data-collection practice is challenged — by a site owner, a regulator, or opposing counsel — the question is rarely “did you write good code?” It is “can you prove what you were permitted to collect, and can you trace every record back to where and when it came from?” Answering both requires two artifacts your pipeline must produce as a byproduct of running: an authorization record (what let you scrape) and a lineage record (the provenance of each datum). This page, a detail under the Mapping Terms of Service for Scrapers technique in the Compliance & Ethical Crawling Foundations section, shows how to capture both for legal defensibility and audits.

Problem Framing #

Authorization and lineage answer different questions and fail in different ways. Authorization is the basis — a signed data-sharing agreement, an API key issued under documented terms, a recorded acceptance of a ToS, or a documented legitimate interest. Lineage is the chain of custody — for a given field in your warehouse, which URL produced it, at what timestamp, under which robots.txt decision, and through which transforms.

The lineage chain one stored field must carryAny break in the chain makes the field impossible to defend or delete.The lineage chain one stored field must carry1Authorisationrecord2Fetch eventwith agent3Parse ruleand version4Stored fieldwith hashes
Any break in the chain makes the field impossible to defend or delete.

The failure modes are asymmetric. Missing authorization is an existential legal problem: without it, a scrape can be characterised as exceeding authorised access, which is exactly the terrain the Understanding CFAA implications for web scraping page covers. Missing lineage is an evidential problem: even if you were authorised, you cannot prove a specific record was collected within scope, cannot honour a deletion request scoped to a source, and cannot reproduce a dataset for audit. Both need to be captured at collection time, because reconstructing them months later from raw dumps is unreliable and self-serving.

Step-by-Step Implementation #

  1. Create an immutable authorization record per source before the first request, capturing the basis, its evidence, and its validity window.
  2. Snapshot the terms you relied on — hash the ToS/API-terms text so you can prove which version was in force when you collected.
  3. Stamp every fetched response with a lineage envelope: source URL, canonical URL, fetch timestamp, HTTP status, the robots.txt allow/deny decision, and the authorization ID that covered it.
  4. Append a transform trail as the record moves through parsing, normalisation, and enrichment, so each mutation is attributable.
  5. Persist lineage to append-only storage separate from the mutable data warehouse, and ship it alongside your operational logs.
Lineage columns worth carrying on every rowFive columns cover both an audit request and an erasure request.Lineage columns worth carrying on every rowsource_urlexact URL fetched, after redirectsfetched_atUTC timestamp of the responsepolicy_hashhash of the rules snapshot in forceextractor_versionthe parser build that produced the fieldlawful_basisthe recorded justification for holding it
Five columns cover both an audit request and an erasure request.
import hashlib
import json
import uuid
from datetime import datetime, timezone

def _now() -> str:
    return datetime.now(timezone.utc).isoformat()

def authorization_record(source: str, basis: str, terms_text: str,
                         valid_until: str, evidence_uri: str) -> dict:
    """One immutable row per source describing WHY we may scrape it."""
    return {
        "auth_id": f"auth-{uuid.uuid4().hex[:12]}",
        "source": source,
        "basis": basis,  # e.g. "api_terms" | "written_permission" | "legitimate_interest"
        "terms_sha256": hashlib.sha256(terms_text.encode()).hexdigest(),
        "evidence_uri": evidence_uri,  # signed PDF, email thread, contract in object store
        "recorded_at": _now(),
        "valid_until": valid_until,
    }

def lineage_envelope(response, canonical_url: str, robots_decision: str,
                     auth_id: str) -> dict:
    """Provenance stamped onto every fetched resource."""
    body = response.content
    return {
        "lineage_id": f"lin-{uuid.uuid4().hex[:16]}",
        "source_url": response.url,
        "canonical_url": canonical_url,
        "fetched_at": _now(),
        "http_status": response.status_code,
        "content_sha256": hashlib.sha256(body).hexdigest(),
        "robots_decision": robots_decision,  # "allow" | "disallow" | "no_robots"
        "auth_id": auth_id,                   # ties this fetch to its authorization
        "transforms": [],                     # appended as the record is processed
    }

def record_transform(envelope: dict, stage: str, detail: str) -> dict:
    envelope["transforms"].append({"stage": stage, "detail": detail, "at": _now()})
    return envelope

A completed lineage envelope for one record is a self-contained audit exhibit:

{
  "lineage_id": "lin-9f2c4a1b7e0d3c88",
  "source_url": "https://example.com/directory/jane-doe",
  "canonical_url": "https://example.com/directory/jane-doe",
  "fetched_at": "2026-07-05T09:14:22+00:00",
  "http_status": 200,
  "content_sha256": "d2f7...",
  "robots_decision": "allow",
  "auth_id": "auth-3b1f9c0a2d55",
  "transforms": [
    {"stage": "parse", "detail": "extracted name,email via css .contact", "at": "2026-07-05T09:14:23+00:00"},
    {"stage": "normalise", "detail": "email lowercased, pseudonymised", "at": "2026-07-05T09:14:23+00:00"}
  ]
}

Verification & Testing #

Prove that (a) no record can be persisted without a valid authorization, and (b) lineage round-trips faithfully.

Lineage assertions a nightly job can runThe failing check is the one that turns into a deletion job.Lineage assertions a nightly job can run✓Every row resolves to an authorisation record✓No row references a policy hash that is unknown✓Extractor versions in the table still exist in the registry✗Rows past their retention window are already gone
The failing check is the one that turns into a deletion job.
def test_every_record_carries_a_valid_auth():
    auth = authorization_record(
        "example.com", "api_terms", "TERMS v3 ...",
        valid_until="2027-01-01T00:00:00Z", evidence_uri="s3://legal/example-terms-v3.pdf")
    env = lineage_envelope(_FakeResp("https://example.com/x", 200, b"<html>"),
                           "https://example.com/x", "allow", auth["auth_id"])
    assert env["auth_id"] == auth["auth_id"]
    # A fetch with no matching authorization must be rejected upstream.
    assert env["robots_decision"] in {"allow", "disallow", "no_robots"}

def test_terms_hash_is_stable_and_detects_change():
    a = authorization_record("s", "api_terms", "TERMS A", "2027-01-01T00:00:00Z", "uri")
    b = authorization_record("s", "api_terms", "TERMS A (edited)", "2027-01-01T00:00:00Z", "uri")
    assert a["terms_sha256"] != b["terms_sha256"]  # any wording change is provable

Operationally, a lineage store should let you answer, in one query, “show every record sourced from example.com between two dates whose robots_decision was disallow” — a non-empty result is an incident, and the ability to run the query at all is the audit capability you are building.

Compliance & Operational Guardrails #

  • Authorization must predate collection. A record created after a dispute begins is worth little. Timestamp it, store the underlying evidence (signed agreement, API key issuance, ToS acceptance) in immutable object storage, and reference it by auth_id.
  • Hash the terms you relied on. ToS pages change silently; a terms_sha256 lets you prove which version governed a given fetch, complementing the versioned-snapshot approach in the parent ToS mapping technique.
  • Keep lineage append-only and separate. Ship it to the structured logging and log shipping pipeline so provenance survives even if the warehouse row is later deleted for a right-to-erasure request — you retain proof of what you did without retaining the personal data itself.
  • Bind lineage to legal exposure. The robots_decision and auth_id fields are precisely the facts that determine whether access was authorised, which is why they belong in the same envelope rather than in scattered logs.

Making Lineage Survive Transformation #

Lineage columns are easy to attach at ingest and easy to lose at the first aggregation. A join that keeps only the left table’s metadata, a GROUP BY that collapses ten source rows into one, a deduplication pass that keeps an arbitrary survivor — each silently severs the link between a stored value and the fetch that produced it, and each is discovered only when someone asks where a number came from.

Three conventions keep the chain intact through transformation:

Carry an array, not a scalar. When rows are merged, the surviving record should hold every contributing source_url and fetched_at, not just one. Storage cost is trivial next to the cost of being unable to answer an erasure request.

Propagate deliberately in every join. Make lineage propagation part of the transformation contract rather than a convention: a derived table that lacks the lineage columns should fail its own tests, not merely look tidy.

Version the transformation. Record which pipeline build produced a derived row, so a bug found later can be scoped to the rows it actually affected rather than to the whole table.

-- Merging duplicates while preserving every contributing source.
INSERT INTO listings_curated AS c (
    natural_key, title, price_minor, currency,
    source_urls, first_fetched_at, last_fetched_at,
    policy_hashes, extractor_version, pipeline_version
)
SELECT
    s.natural_key,
    (array_agg(s.title ORDER BY s.fetched_at DESC))[1],
    (array_agg(s.price_minor ORDER BY s.fetched_at DESC))[1],
    (array_agg(s.currency ORDER BY s.fetched_at DESC))[1],
    array_agg(DISTINCT s.source_url),
    min(s.fetched_at),
    max(s.fetched_at),
    array_agg(DISTINCT s.policy_hash),
    max(s.extractor_version),
    :pipeline_version
FROM listings_staging s
GROUP BY s.natural_key
ON CONFLICT (natural_key) DO UPDATE SET
    source_urls    = (SELECT array_agg(DISTINCT u)
                      FROM unnest(c.source_urls || EXCLUDED.source_urls) AS u),
    last_fetched_at = greatest(c.last_fetched_at, EXCLUDED.last_fetched_at),
    policy_hashes   = (SELECT array_agg(DISTINCT h)
                       FROM unnest(c.policy_hashes || EXCLUDED.policy_hashes) AS h);

The union-on-conflict pattern is what makes a recrawl additive rather than destructive: the curated row accumulates the evidence of every fetch that contributed to it, while the field values reflect the most recent observation. That single property is what lets a later deletion request find and remove every source of a record, which is exactly what the right-to-erasure workflow depends on.

A Nightly Lineage Audit #

Lineage that is never checked drifts. A short nightly job comparing the curated table against the authorisation registry catches the three failures that matter: rows referencing a host with no current registry entry, rows carrying a policy hash that no longer exists in the snapshot store, and rows past their retention date. The first two indicate a broken chain, the third indicates a control that did not run — and all three are far cheaper to find on a Tuesday than during a review.

Common Mistakes #

  1. Relying on ambient logs as lineage. Web-server access logs record that a request happened, not why it was authorised or how the resulting data was transformed. Lineage must be a first-class, structured artifact attached to the record.
  2. Capturing authorization once and never revalidating. A valid_until that has passed, or a ToS hash that no longer matches the live page, means your basis has lapsed. Re-check on a schedule and block sources whose authorization has expired.
  3. Storing lineage in the same mutable table as the data. When you delete a record for erasure or dedup, you erase its provenance too. Keep the audit trail in append-only storage so you can prove past conduct without hoarding the underlying personal data.

Frequently Asked Questions #

What counts as adequate authorization to scrape a site? #

It depends on the source, but strong forms include a signed data-sharing or licensing agreement, an API key issued under documented terms you accepted, or a recorded acceptance of a clickwrap ToS that permits your use. A documented legitimate-interest assessment can support collection of public data. Weak or absent authorization — scraping in the face of a Disallow or an explicit prohibition — is what turns a technical activity into a legal exposure, so record the basis and its evidence before you start.

Why hash the terms of service instead of just saving a copy? #

Do both: store the copy and record its SHA-256. The hash gives you a compact, tamper-evident fingerprint that proves which exact wording was live when you collected, so if the publisher later edits the ToS you can show what you actually agreed to. Comparing a fresh hash against the stored one is also how you detect that the terms changed and your authorization needs re-review.

How does data lineage help with a GDPR erasure request? #

Lineage lets you locate every record derived from a given source or subject and the transforms applied to each, so you can delete precisely the right rows and prove you did. Because the lineage envelope is stored append-only and separately from the data, you can retain evidence that you handled the request correctly — the lineage_id, timestamps, and decision — without retaining the personal data you were asked to erase.