Automating GDPR right-to-erasure deletions

When a data subject exercises their right to erasure under Article 17 of the GDPR, “we deleted it” is not a defensible answer — you must locate every copy of their data across every sink, remove it, and be able to prove you did. For a scraping pipeline that fans records out to Postgres, Parquet on object storage, a search index, and log archives, that is an engineering problem, not a manual one. This page describes an automated erasure workflow: a request queue, cross-sink subject location, hard delete plus tombstone, verification, and an immutable audit log. It is a detail page under Data Retention and GDPR Deletion Workflows, part of the Pipeline Storage, Deduplication & Monitoring section.

Problem Framing #

Article 17 gives a subject the right to have their personal data erased “without undue delay”, which regulators read as within one month. The obligation is not satisfied by deleting the row a user can see in your app — it extends to every derived copy: warehouse tables, columnar dumps, caches, search indices, and backups. A pipeline that continuously ingests scraped personal data multiplies these copies, so erasure must be systematic and repeatable.

Resolution is the hard step: it needs an index built before the request arrives.

The design tension is between a hard delete (the data is gone) and proof (evidence it was gone, retained for accountability under Article 5(2)). You cannot keep the personal data to prove you deleted it. The resolution is a tombstone: a record keyed by a non-identifying subject reference that says “subject X was erased at time T across sinks Y”, holding no personal data itself, plus an append-only audit log. The tombstone also prevents resurrection — a later re-crawl of the same source must be blocked from re-inserting the erased subject.

Step-by-Step Implementation #

Queue the request. Persist every erasure request with a status and deadline so nothing is lost and SLA breaches are visible.
Resolve the subject to keys. Map the request (an email, a profile URL) to the pseudonymous subject_ref your records are keyed by — never scan raw personal data across sinks.
Fan out hard deletes to each registered sink through a common interface, collecting per-sink row counts.
Write a tombstone so re-crawls cannot resurrect the subject.
Verify that zero records remain in each sink, then close the request.
Append to the audit log — who requested, what was deleted, when, and the verification result.

Without this index, erasure becomes a full scan of every store you own.

import hashlib
from datetime import datetime, timezone, timedelta

def subject_ref(identifier: str) -> str:
    # Keyed pseudonym: what records are indexed by, not the raw identifier.
    return hashlib.sha256(("subj:" + identifier.lower().strip()).encode()).hexdigest()

class ErasureWorkflow:
    def __init__(self, sinks: list, audit, tombstones):
        self.sinks = sinks            # each implements delete(ref) + count(ref)
        self.audit = audit
        self.tombstones = tombstones

    def enqueue(self, conn, identifier: str, requested_by: str) -> str:
        ref = subject_ref(identifier)
        deadline = datetime.now(timezone.utc) + timedelta(days=30)  # Art. 12(3)
        with conn.cursor() as cur:
            cur.execute(
                "INSERT INTO erasure_request (subject_ref, requested_by, status, deadline)"
                " VALUES (%s, %s, 'pending', %s)"
                " ON CONFLICT (subject_ref) DO NOTHING",
                (ref, requested_by, deadline),
            )
        conn.commit()
        return ref

    def execute(self, conn, ref: str) -> None:
        results = {}
        for sink in self.sinks:
            deleted = sink.delete(ref)          # hard delete, returns row count
            results[sink.name] = deleted

        # Tombstone holds NO personal data — only the pseudonymous ref + when.
        self.tombstones.add(ref, erased_at=datetime.now(timezone.utc))

        remaining = {s.name: s.count(ref) for s in self.sinks}
        if any(remaining.values()):
            raise AssertionError(f"erasure incomplete: {remaining}")

        with conn.cursor() as cur:
            cur.execute(
                "UPDATE erasure_request SET status='completed', "
                "completed_at=now() WHERE subject_ref=%s", (ref,))
        conn.commit()
        # Append-only: records the act of deletion without the deleted data.
        self.audit.append(action="erasure", subject_ref=ref,
                          deleted=results, verified=remaining)

Each sink implements the same tiny contract so the workflow stays sink-agnostic. Row stores hard-delete by the pseudonymous key; Parquet cannot mutate a file, so erasure means rewriting the affected partition without the subject’s rows:

class PostgresSink:
    name = "postgres"
    def delete(self, ref: str) -> int:
        with self.conn.cursor() as cur:
            cur.execute("DELETE FROM record WHERE subject_ref = %s", (ref,))
            return cur.rowcount
    def count(self, ref: str) -> int:
        with self.conn.cursor() as cur:
            cur.execute("SELECT count(*) FROM record WHERE subject_ref = %s", (ref,))
            return cur.fetchone()[0]

class ParquetSink:
    name = "parquet"
    def delete(self, ref: str) -> int:
        # Immutable files: read the partition, drop matching rows, atomically
        # rewrite it. Partitioning by a low-cardinality key keeps this cheap.
        return rewrite_partition_excluding(self.dataset, subject_ref=ref)

Because records are keyed by subject_ref rather than raw identifiers, locating a subject is an index lookup in each sink, not a scan of personal data. That keying decision should be made at write time, following GDPR compliance for scraped personal data.

Verification & Testing #

Verification is part of the workflow, not an afterthought — the execute method above already re-counts every sink and refuses to mark the request complete if anything remains. Test the full round trip, including resurrection resistance:

The backup sweep is the step most workflows discover only during an incident.

def test_erasure_removes_and_stays_removed(workflow, conn, sinks):
    ref = workflow.enqueue(conn, "[email protected]", requested_by="dpo")
    seed_records(sinks, ref, n=5)

    workflow.execute(conn, ref)
    assert all(s.count(ref) == 0 for s in sinks)          # gone everywhere

    # A later re-crawl must NOT resurrect the subject.
    ingest_scraped_record(sinks, subject_ref=ref, tombstones=workflow.tombstones)
    assert all(s.count(ref) == 0 for s in sinks)          # still gone

For an operational check, query the request queue for anything past its deadline: SELECT subject_ref, deadline FROM erasure_request WHERE status='pending' AND deadline < now() should always return zero rows. Wire that into a Prometheus alert so a stalled erasure is visible before it becomes a regulatory breach.

Compliance & Operational Guardrails #

The tombstone must contain no personal data. It exists to prove erasure and to block re-ingestion; storing the erased identifier in it would recreate the very data you deleted. Key it on the pseudonymous subject_ref only.
Address backups explicitly. Regulators accept that erasing from immutable backups on a rolling schedule is proportionate — document the backup rotation window and re-apply the tombstone if a restore reintroduces the subject.
Keep the audit log append-only and access-controlled. Article 5(2) accountability requires evidence you can produce on demand; a mutable log undermines it.
Honour the one-month deadline. Track deadline per request and alert before it lapses; extensions under Article 12(3) must themselves be logged.

Resolving a Person to Rows #

The step that determines whether an erasure workflow is a script or a project is resolution: turning “this person” into “these rows, in these stores”. Doing it by scanning free text at request time is slow, incomplete, and impossible to prove correct. Doing it from an index maintained at write time is a query.

import hashlib
import hmac

def subject_key(identifier: str, salt: bytes) -> bytes:
    """Stable, non-reversible key for a normalised identifier (email, handle, phone)."""
    normalised = identifier.strip().lower()
    return hmac.new(salt, normalised.encode("utf-8"), hashlib.sha256).digest()

def link_on_write(conn, identifiers: list[str], store: str, record_id: str, salt: bytes) -> None:
    """Called INSIDE the same transaction that writes the record."""
    conn.executemany(
        """INSERT INTO subject_index (subject_key, store_name, record_id)
           VALUES (%s, %s, %s) ON CONFLICT DO NOTHING""",
        [(subject_key(i, salt), store, record_id) for i in identifiers],
    )

def resolve(conn, identifier: str, salt: bytes) -> list[tuple[str, str]]:
    return conn.execute(
        "SELECT store_name, record_id FROM subject_index WHERE subject_key = %s",
        (subject_key(identifier, salt),),
    ).fetchall()

Using an HMAC rather than a bare hash is what keeps the index from becoming a lookup table for anyone who obtains it: without the salt, a candidate identifier cannot be tested against it. Keep the salt in a secret store, not in configuration, and treat rotating it as a migration rather than a routine change.

Normalise identifiers consistently on both the write and the resolve path — the same case folding, the same whitespace handling, the same treatment of address subaddressing — or a request will resolve to nothing while the rows plainly exist. Test that with a fixture that writes a record under one spelling and resolves it under another.

The Confirmation, and What It Should Say #

An erasure workflow ends with a statement to the requester, and the statement should be specific enough to be verifiable without disclosing anything new. Name what was deleted in categories rather than in detail, name the stores in general terms, state the date of completion, and state what was retained and why — because something usually is. Audit records demonstrating that the deletion occurred are themselves retained, and so, for a bounded window, are backups.

Record the confirmation alongside the deletion audit entry, so the chain from request to resolution to deletion to reply is one queryable sequence. If the same person contacts you again, that chain is the answer. Where the deletion could not be completed within the statutory window, the record of why is what turns a breach into a documented delay.

Common Mistakes #

Soft-deleting when erasure is required. A deleted=true flag leaves the personal data intact and does not satisfy Article 17. Hard-delete the data; keep only a non-identifying tombstone.
Forgetting derived sinks. Deleting from Postgres but leaving copies in Parquet, a search index, or logs means the data is not erased. Register every sink behind one interface.
No resurrection guard. Without a tombstone, the next crawl re-inserts the subject and silently undoes the erasure.

Frequently Asked Questions #

Do I have to delete the subject’s data from backups too? #

Yes, the right to erasure covers backups, but regulators accept a proportionate approach: if backups are immutable and rotate on a defined schedule, you may let the erased data expire with the normal rotation rather than restoring and rewriting every archive. Document the rotation window, and if you restore a backup that predates an erasure, re-run the tombstone check so the subject is removed again before the data returns to service.

How does a tombstone let me prove deletion without keeping the personal data? #

The tombstone stores only a keyed pseudonym (subject_ref), the erasure timestamp, and which sinks were cleared — none of which is personal data on its own. Combined with an append-only audit log, it demonstrates that a specific request was fulfilled and when, satisfying the Article 5(2) accountability duty, while the underlying name, email, or address is genuinely gone.

Data Retention and GDPR Deletion Workflows — the parent technique covering retention schedules and lawful deletion.
Deduplicating records with content hashing — how records are keyed and fingerprinted before they reach a sink.
GDPR compliance for scraped personal data — the write-time obligations that make erasure feasible later.