GDPR Compliance for Scraped Personal Data #

The moment a crawler ingests a name, an email address, a profile URL, or even an IP tied to a natural person, it crosses from “collecting documents” into “processing personal data” under the General Data Protection Regulation — and inherits obligations that no amount of clever request throttling will discharge. This technique page, part of the Compliance & Ethical Crawling Foundations section, walks through how to decide when scraped data is regulated, how to establish a lawful basis before the first request, and how to bake minimisation, transparency, and data-subject rights into the pipeline rather than bolting them on after an audit. It is written for engineers who own the ingest path and the compliance officers who have to defend it.

Core Principles & Regulatory Foundation #

GDPR bites on processing of personal data about an identified or identifiable natural person (a data subject). Scraping is processing. Storing is processing. Deleting is processing. The threshold for “personal data” under Article 4(1) is deliberately low: anything that can single out an individual, directly or in combination with other data you hold, qualifies. A publicly visible email on a company contacts page is still personal data. So is a pseudonymous forum handle if you can link it back to a person.

Publication does not neuter the regulation. The Weltimmo and Ryneš line of CJEU cases, and the regulators’ consistent guidance, make clear that “the data was public” is not a lawful basis and not an exemption. You still need a legal ground under Article 6, transparency under Articles 13–14, and you still owe data subjects their rights.

When scraped data is “personal data” #

Use this table to classify fields at the point of extraction, not after storage. The classification drives every downstream control.

Article 9 special-category data (health, ethnicity, political opinion, religion, sexual orientation, biometric, trade-union membership) is effectively prohibited to process unless a narrow exception applies. Legitimate interest is not one of those exceptions. If your crawl target is a health forum or a political community, assume you cannot scrape identifiable member data at all without explicit consent, which is impractical to obtain — so exclude those sources.

Lawful basis: legitimate interest vs consent #

Consent under GDPR must be freely given, specific, informed, and unambiguous. You cannot obtain valid consent from someone whose page you scraped without their knowledge, so consent is almost never the workable basis for scraping. That leaves legitimate interest under Article 6(1)(f) as the realistic ground for most compliant scraping of business or public-professional data.

Legitimate interest is not a free pass. It requires a documented three-part Legitimate Interests Assessment (LIA):

1. Purpose test — is there a genuine, lawful, specifically articulated interest? (“Building a B2B sales-lead dataset” is specific; “collecting data that might be useful” is not.)
2. Necessity test — is scraping necessary, or could you achieve the purpose with less intrusive means (a licensed dataset, an API, aggregate data)?
3. Balancing test — do the individual’s rights and reasonable expectations override your interest? Someone posting a work email expects business contact; they do not expect enrolment in an automated profiling system.

Record the LIA outcome as a versioned artifact tied to the crawl configuration, and make the balancing test fail closed for anything touching Article 9 data or children’s data.

Implementation Steps #

The controls below turn the legal analysis into pipeline code. Classify, minimise, and log at ingest — before personal data is ever persisted in a warehouse.

Step 1 — Classify and tag fields at extraction #

Attach a data-classification label to every extracted field so downstream stages (storage, retention, erasure) can act on it without re-parsing raw HTML.

from dataclasses import dataclass, field
from enum import Enum

class DataClass(Enum):
    NON_PERSONAL = "non_personal"
    PERSONAL = "personal"          # Art. 4 personal data
    SPECIAL = "special_category"   # Art. 9 — near-prohibited

@dataclass
class ExtractedField:
    name: str
    value: str
    data_class: DataClass
    lawful_basis: str | None = None  # e.g. "legitimate_interest:lia-2026-014"

SPECIAL_KEYWORDS = {"health", "diagnosis", "religion", "union", "orientation"}

def classify(name: str, value: str) -> DataClass:
    lname = name.lower()
    if any(k in lname for k in ("email", "name", "phone", "profile", "ip")):
        # Generic role addresses are not about a natural person
        if lname == "email" and value.split("@")[0] in {"info", "contact", "sales"}:
            return DataClass.NON_PERSONAL
        return DataClass.PERSONAL
    if any(k in lname for k in SPECIAL_KEYWORDS):
        return DataClass.SPECIAL
    return DataClass.NON_PERSONAL

If classify returns DataClass.SPECIAL, the correct behaviour is to drop the field (or the whole record) and emit a compliance event — do not persist it pending review.

Step 2 — Minimise and pseudonymise before persistence #

Article 5(1)© (data minimisation) means you store only fields the documented purpose actually needs. Article 32 encourages pseudonymisation as a security-of-processing measure. Do both at the ingest boundary so raw identifiers never hit durable storage in cleartext where they are not required.

import hashlib
import hmac

# Pepper stored in a secrets manager, NOT in code or the DB.
_PEPPER = load_secret("PSEUDONYM_PEPPER")

def pseudonymise(value: str) -> str:
    # Keyed hash so the mapping is not reversible by rainbow tables,
    # and can be rotated/retired to effect erasure.
    return hmac.new(_PEPPER, value.encode(), hashlib.sha256).hexdigest()

def minimise(record: dict, needed: set[str]) -> dict:
    return {k: v for k, v in record.items() if k in needed}

Keep the pseudonymisation key separate from the pseudonymised data. Destroying the key is itself a valid route to rendering data effectively anonymous, which connects directly to your deletion strategy in the data retention and GDPR deletion workflows technique.

Step 3 — Satisfy Article 14 transparency (data not obtained from the subject) #

Because you collect from a third-party site rather than from the person, Article 14 applies: you must proactively inform data subjects — normally within one month — of your identity, purposes, legal basis (your legitimate interest), the categories of data, recipients, retention period, and their rights. There is a narrow “disproportionate effort” exemption under Art. 14(5)(b), but regulators (see the Polish DPA’s Bisnode fine) expect you to at least publish a prominent privacy notice and, where you hold contact details, to notify directly.

def article14_notice_targets(records: list[dict]) -> list[str]:
    # Where we hold a deliverable contact address, direct notice is expected.
    return [r["email"] for r in records
            if r.get("email") and r.get("data_class") == "personal"]

Maintain a public, versioned privacy notice URL and record which notice version applied to each ingest batch, so you can prove what a data subject was told and when.

Step 4 — Run a DPIA when profiling or scale demands it #

A Data Protection Impact Assessment (Article 35) is mandatory when processing is likely to result in high risk — which explicitly includes systematic evaluation/profiling and large-scale processing. Bulk scraping of personal data to build profiles is a textbook DPIA trigger. Treat the DPIA as a gated CI artifact: no new personal-data source ships without one.

• Data source and field inventory linked to classification labels
• Documented lawful basis + completed LIA for each personal field
• Necessity and proportionality assessment vs. less intrusive alternatives
• Special-category and children's data explicitly excluded or justified
• Retention period defined and enforced by TTL, not left indefinite
• Article 14 notice drafted, published, and version-pinned to the crawl
• Data-subject-rights runbook wired to the storage layer
• Residual-risk sign-off recorded (and DPO consulted where appointed)

Error Handling & Observability #

Compliance failures are silent by default — a mis-classified special-category field persists happily until a regulator asks. Make the pipeline loud about the states that matter, and keep an immutable audit trail. Route these events into the structured logging and log shipping pipeline so the audit record lives outside the application database and survives its deletion.

{
  "event": "gdpr.ingest.record",
  "trace_id": "b8f2...",
  "source_url": "https://example.com/team",
  "fields_personal": 3,
  "fields_special_dropped": 1,
  "lawful_basis": "legitimate_interest:lia-2026-014",
  "notice_version": "privacy-2026-06",
  "retention_expires": "2027-01-05T00:00:00Z"
}

Useful Prometheus signals: gdpr_special_category_dropped_total (alert on any non-zero rate — it means you are crawling the wrong source), gdpr_records_without_basis_total (alert > 0), and gdpr_erasure_pending_seconds (alert when the oldest open request approaches the 30-day SLA).

Compliance Boundaries #

GDPR does not stand alone: the same crawl may implicate the ePrivacy rules on cookies, national implementations with stricter carve-outs, and — for the CFAA and contractual angle — the site’s own terms, which is why lawful-basis work sits alongside documenting scraping authorization and data lineage. The engineering guardrails:

• Every personal field carries a lawful basis before it is persisted
• Special-category and children's data are excluded at the source level
• Retention is finite, enforced, and shorter than "forever"
• Data-subject access, rectification, objection, and erasure are all serviceable
• The right to object to Art. 6(1)(f) processing is honoured without friction

A quick scale check: if a source yields 50,000 personal records and your Article 14 obligation is direct notice where you hold an email, budget for the send volume and the objection-handling load before you run the crawl — an unservable rights backlog is itself a violation.

Recognising Personal Data You Did Not Mean to Collect #

The fields a pipeline intends to collect are rarely the problem. The problem is the personal data that arrives incidentally: an author byline in a product review, a phone number inside a free-text listing description, a photographer’s name in an image caption, a username embedded in a profile URL. None of these appear in the schema, all of them are personal data, and all of them are subject to the same obligations as a field you deliberately extracted.

Two controls catch this at ingest rather than during an audit. The first is a declared-field policy: the sink accepts only fields the schema names, and anything else is either dropped or routed to a review queue rather than stored opportunistically. The second is a scanning pass over free-text fields that flags likely identifiers before the record is written.

import re

PATTERNS = {
    "email":  re.compile(r"[\w.+-]+@[\w-]+\.[\w.]{2,}"),
    "phone":  re.compile(r"(?<!\d)(?:\+\d{1,3}[ -]?)?(?:\d[ -]?){9,13}\d(?!\d)"),
    "iban":   re.compile(r"\b[A-Z]{2}\d{2}[A-Z0-9]{10,30}\b"),
    "handle": re.compile(r"(?<!\w)@[A-Za-z0-9_]{3,30}(?!\w)"),
}

def scan_free_text(value: str) -> list[str]:
    """Return the kinds of identifier found in a free-text field."""
    return [kind for kind, rx in PATTERNS.items() if rx.search(value or "")]

Regular expressions are a coarse instrument and will produce false positives — a product code can look like a phone number, a date range like an account identifier. That is acceptable, because the output is a routing decision rather than a verdict: flagged records go to a queue where a person confirms the classification and, more importantly, decides whether the selector should have captured that text in the first place. Most incidental personal data is best fixed upstream by narrowing the extraction, not downstream by redacting the output. Where the field is genuinely needed, pseudonymising scraped personal data at ingest describes how to keep it usable without keeping it identifying.

Transparency When There Is No Direct Contact #

Collecting personal data from a website means you rarely have a channel to the data subject, which does not remove the obligation to inform them. The workable pattern is a public processing notice: a stable page describing what categories of personal data you collect, from which kinds of source, for what purpose, on what lawful basis, how long you keep it, and how to object or request erasure. Link it from the same crawler information page that your crawler token points at, so a person who finds your crawler in a site’s access log can reach it in one hop.

Record the notice’s publication date and content hash the same way the terms mapping records a site’s terms. When the categories you collect change, the notice must change too, and being able to show which version was live on a given date is what makes the transparency claim verifiable rather than asserted.

Common Mistakes #

1. “It was public, so GDPR doesn’t apply.” Wrong. Public availability changes the balancing test slightly; it never removes the need for a lawful basis, transparency, or rights handling.
2. Defaulting to consent as the lawful basis. You cannot get valid consent from someone you scraped silently. Use — and document — legitimate interest instead, and accept that some sources fail the balancing test.
3. Skipping Article 14 because “notifying everyone is too hard.” The disproportionate-effort exemption is narrow and was tested against a real fine. At minimum publish a versioned notice and directly notify where you hold contact details.
4. Storing raw identifiers indefinitely. No retention limit breaches Art. 5(1)(e). Set a TTL at ingest and enforce it with an automated deletion job.
5. Treating pseudonymisation as anonymisation. Keyed-hash data is still personal data if you (or anyone) can reverse or re-link it. Only irreversible severing — e.g. destroying the key — approaches anonymity.

Frequently Asked Questions #

Is scraping publicly available personal data legal under GDPR? #

It can be, but “public” is not itself a lawful basis. You still need a valid Article 6 ground — in practice a documented legitimate interest that passes a three-part balancing test — plus Article 14 transparency and working data-subject rights. If the balancing test favours the individual, or the data is special-category, the answer is no.

Do I have to tell people I scraped their data? #

Yes, in most cases. Article 14 governs data not collected from the subject and requires you to provide your identity, purposes, legal basis, retention, and their rights, normally within a month. There is a narrow disproportionate-effort exemption, but regulators expect a prominent published privacy notice and direct notification where you hold usable contact details.

Can legitimate interest cover scraping special-category data like health or political views? #

No. Article 9 special-category data is near-prohibited and legitimate interest is not one of the listed exceptions. The only realistic ground is explicit consent, which you cannot obtain by silent scraping. Exclude such sources at the source level rather than filtering after the fact.

How does pseudonymisation help my GDPR posture if the data is still personal? #

Pseudonymisation is a security-of-processing measure under Article 32 that reduces risk and can strengthen your legitimate-interest balancing, but pseudonymised data remains personal data because it is re-linkable. Its real leverage comes later: destroying the keyed-hash pepper is a clean, verifiable way to render whole cohorts of records effectively irreversible, supporting bulk erasure and retention enforcement.

When is a DPIA mandatory for a scraping project? #

Whenever processing is likely to be high-risk — which Article 35 and the regulators tie explicitly to large-scale processing and systematic profiling. Bulk collection of personal data to build or enrich profiles hits both criteria, so treat the DPIA as a required, version-controlled gate that must be signed off before a new personal-data source goes live.

Related guides #

• Compliance & Ethical Crawling Foundations — the parent section covering the legal and technical basis for polite, defensible crawling.
• Mapping Terms of Service for Scrapers — how contractual limits layer on top of statutory ones like GDPR.
• Data Retention and GDPR Deletion Workflows — enforcing finite retention and the deletion machinery this page assumes.
• Automating GDPR Right-to-Erasure Deletions — servicing erasure requests across a warehouse within the statutory SLA.
• Structured Logging and Log Shipping — shipping the compliance audit trail to durable, tamper-evident storage.