Pipeline Storage, Deduplication & Monitoring #

Extraction is only the middle of a scraping pipeline; the durable engineering happens afterward, when a validated record has to land somewhere queryable, survive re-crawls without duplicating, honour a deletion request months later, and stay observable while thousands of pages flow through. This section is the operational backbone for that stage: where scraped data goes once it leaves the parser, how you keep it clean and lawful over time, and how you watch the crawl that produces it. It is written for the data engineers who own the warehouse, the full-stack developers wiring up sinks, the researchers who need reproducible datasets, and the compliance officers who have to answer for what is stored and for how long. Every decision here sits on two axes at once — technical correctness (idempotent writes, partitioning, alerting) and legal defensibility (retention limits, erasure, minimisation) — and the two are not separable. A sink that cannot delete a person is a compliance liability no matter how fast it ingests.

The upstream stages feed directly into this one. The Data Parsing & Transformation Pipelines section produces the typed records that arrive here; a well-defined Pydantic schema gate is the contract that lets a storage layer trust its inputs. The Network Resilience & Proxy Management section governs how the crawl reaches pages, and the metrics it emits — retry counts, proxy health, queue depth — are consumed by the observability tooling described here. And the Compliance & Ethical Crawling Foundations section sets the legal frame — ToS mapping and GDPR handling of scraped personal data — that dictates what you are even allowed to persist. Storage is where all four concerns become concrete and auditable.

Core Principles of Durable, Lawful Storage #

Storage is a compliance surface, not a bucket #

The moment scraped data is written to a durable sink, you have created records that a regulator, a data subject, or a litigant can ask about. Under GDPR that means every stored dataset needs a defined lawful basis, a retention period, and a mechanism to locate and delete an individual’s records on request. Under the CFAA and site terms of service, it means you must not be retaining data you were never authorised to collect. The engineering consequence is that “just dump it to S3” is rarely acceptable: you need to know what personal data a row contains, which crawl produced it, and when it is scheduled to expire. Those requirements shape the schema before you write a single row, which is why data retention and GDPR deletion workflows are treated here as a first-class design concern rather than an afterthought bolted on when a request arrives.

Idempotency over volume #

A resilient crawler re-visits pages, retries failed fetches, and occasionally replays an entire batch after a partial failure. If your sink appends blindly, every one of those events multiplies your row count and corrupts every downstream aggregate. The governing principle is that writing the same logical record twice must produce the same end state as writing it once. That is achieved through natural keys, upserts, and content-addressed identifiers rather than auto-incrementing surrogate keys assigned at insert time. Idempotency also underpins data minimisation: if you can recognise that a record is unchanged since the last crawl, you can skip the write entirely and avoid storing redundant copies of the same personal data.

Separation of raw, staged, and curated tiers #

Mature pipelines keep at least three tiers: an immutable raw landing zone (often compressed JSON or the original HTML), a staged tier of validated typed records, and a curated tier optimised for query. This separation lets you re-derive curated tables when parsing logic changes, quarantine bad payloads without losing them, and apply different retention rules per tier — raw HTML containing PII might expire in 30 days while an anonymised aggregate persists for years. It also makes deletion tractable: erasure workflows target the tiers that hold identifiable data, not the anonymised rollups.

The practical payoff of tiering shows up the first time a parser bug ships. If you kept only the curated tier, a bad extraction rule silently corrupts your dataset and there is nothing to recompute from. With an immutable raw tier, you fix the rule, replay the affected partitions, and the curated tables self-heal — the raw bytes never changed. This is why the tiers are ordered by mutability: raw is append-only and never edited, staged is rebuildable, and only the curated tier is treated as disposable-and-regenerable. Retention rules then attach to whichever tier actually holds identifiable data, so you can be generous with anonymised aggregates and strict with anything that names a person.

Technical Implementation Across the Storage Stage #

Choosing and writing to structured sinks #

The first decision is the shape of the sink. Relational stores like Postgres give you transactional upserts and strong constraints; object storage like S3 gives you cheap, immutable, infinitely scalable landing; columnar formats like Parquet give you compact analytical files; and warehouses like BigQuery or Snowflake give you SQL over the lot. Most production pipelines use several at once — raw to S3, curated to Postgres or a warehouse. The patterns for batching, partitioning, idempotent writes, and schema evolution across all of these are covered in structured data sinks and warehousing.

import psycopg2.extras

def upsert_products(conn, rows: list[dict]) -> int:
    """Idempotent batch upsert keyed on a natural business key."""
    sql = """
        INSERT INTO products (sku, title, price, source_url, crawled_at)
        VALUES %s
        ON CONFLICT (sku) DO UPDATE SET
            title = EXCLUDED.title,
            price = EXCLUDED.price,
            crawled_at = EXCLUDED.crawled_at
        WHERE products.crawled_at < EXCLUDED.crawled_at
    """
    values = [(r["sku"], r["title"], r["price"], r["source_url"], r["crawled_at"])
              for r in rows]
    with conn.cursor() as cur:
        # ON CONFLICT makes replays safe; the WHERE guards against stale overwrites.
        psycopg2.extras.execute_values(cur, sql, values)
        conn.commit()
        return cur.rowcount

Deduplicating before and after the write #

Duplicates enter a pipeline at two levels: the same URL crawled twice, and two different URLs yielding the same content. The first is cheapest to solve at the queue, using a seen-set as described in the distributed crawl scheduling and queues guidance. The second requires content-level detection — exact hashing for byte-identical records, and near-duplicate techniques for pages that differ only in a timestamp or session token. The full toolkit of exact keys, SHA-256 content hashes, SimHash/MinHash near-duplicate detection, and Redis or Bloom-filter seen-sets lives in deduplication strategies for scraped data.

import hashlib

def content_fingerprint(record: dict, fields: tuple[str, ...]) -> str:
    """Stable SHA-256 over the fields that define record identity."""
    canonical = "\x1f".join(str(record.get(f, "")) for f in fields)
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

# Skip the write entirely when the fingerprint is unchanged — this is both a
# performance optimisation and a data-minimisation control.

Retention and erasure as scheduled jobs #

Retention is not a policy document; it is a cron job. Every tier that holds personal data needs an automated sweep that deletes or anonymises records past their retention horizon, plus an on-demand path that satisfies a GDPR erasure request within the statutory window. Building these as first-class, logged, testable jobs — rather than manual DELETE statements run under pressure — is the subject of data retention and GDPR deletion workflows.

Observability wired through every stage #

You cannot operate a crawl you cannot see. Request rates, error ratios, retry storms, queue depth, parse-failure rates, and write throughput all need to be exported as metrics and correlated with structured logs. The metric side — a Prometheus client in the scraper and Grafana dashboards over it — is covered in crawl observability with Prometheus and Grafana. The log side — emitting structured JSON events and shipping them to a searchable backend — is covered in structured logging and log shipping.

from prometheus_client import Counter, Histogram

records_written = Counter(
    "scraper_records_written_total", "Records persisted", ["sink", "status"]
)
write_latency = Histogram(
    "scraper_write_latency_seconds", "Sink write latency", ["sink"]
)

def observed_write(sink: str, rows: list[dict], writer) -> None:
    with write_latency.labels(sink=sink).time():
        try:
            writer(rows)
            records_written.labels(sink=sink, status="ok").inc(len(rows))
        except Exception:
            records_written.labels(sink=sink, status="error").inc(len(rows))
            raise

Risk Mitigation & Pipeline Orchestration #

Storage bugs are silent and expensive: a broken upsert key inflates a table for weeks before anyone notices the skewed metric. Guard the stage with automated gates rather than review.

• Pre-flight schema check. Before any batch write, validate the batch against the same Pydantic model that gated parsing. A CI job should fail the build if a sink’s target schema and the validation model drift apart.
• Idempotency test in CI. Run every writer twice against a fixture and assert the row count and content are identical after the second run. This catches append-instead-of-upsert regressions before they reach production.
• Retention dry-run. The deletion job should support a --dry-run flag that logs what it would delete and its total count; wire an alert if that count exceeds a sane threshold, which usually signals a misconfigured retention window.
• Cross-stage hooks. Emit a correlation ID from the fetcher and thread it through parse, validate, and write so a single stored row can be traced back to the exact request that produced it — essential for both debugging and lineage.

Compliance Auditing & Monitoring #

Every write and every deletion should leave an audit trail. A minimal structured event for the storage stage looks like this:

{
  "event": "record_persisted",
  "ts": "2026-07-05T09:14:22Z",
  "correlation_id": "c8f1-9a2e",
  "sink": "postgres.products",
  "operation": "upsert",
  "record_key": "sku:AZ-4471",
  "contains_pii": false,
  "retention_expires_at": "2026-10-05T00:00:00Z",
  "lawful_basis": "legitimate_interest"
}

Expose operational health as Prometheus metrics — scraper_records_written_total, scraper_write_latency_seconds, scraper_duplicates_skipped_total, scraper_deletion_requests_total — and alert when the duplicate-skip ratio collapses (dedup broke) or the deletion-request backlog grows (erasure SLA at risk). Set a retention policy per table, document its lawful basis, and keep the audit log itself for the period your jurisdiction mandates for processing records, typically longer than the data it describes.

Making a Record Findable Before You Need To Find It #

Almost every hard problem at the storage stage reduces to one question: given some identifier — a URL, a person, a crawl run — can you locate every row derived from it, in every store you operate, without a full scan? Teams that answer this at design time handle erasure requests, incident cleanups and reprocessing as routine jobs. Teams that answer it after the first request discover that the answer is “no”, and that fixing it means rebuilding indexes across several systems under time pressure.

Three indexes cover the realistic cases, and all three are cheap to maintain at write time and expensive to construct retrospectively.

-- 1. By source: which rows came from this URL?
CREATE INDEX listings_source_url_idx ON listings_curated (source_url);

-- 2. By run: which rows did this crawl generation produce?
CREATE INDEX listings_generation_idx ON listings_curated (crawl_generation, fetched_at);

-- 3. By subject: which rows reference this person? (hashed, never the raw value)
CREATE TABLE subject_index (
    subject_key   bytea       NOT NULL,       -- sha256(normalised identifier + salt)
    store_name    text        NOT NULL,
    record_id     text        NOT NULL,
    linked_at     timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (subject_key, store_name, record_id)
);
CREATE INDEX subject_index_key_idx ON subject_index (subject_key);

The subject index deserves the most attention because it is the one that cannot be reconstructed later. Once a record has been written and the free-text field it appeared in has been transformed, aggregated or partially redacted, there is no reliable way to determine that it ever referenced a particular person. Writing the index entry at the same moment as the record — inside the same transaction — is the only approach that is correct by construction.

Salt the hash and keep the salt in a secret store. An unsalted hash of an email address is trivially reversible against a dictionary of addresses, which converts a privacy control into a privacy problem. Rotate deliberately and rarely, because a rotation invalidates every existing entry and requires a rebuild from the source records.

Every Store, Not Just the Database #

The index is only useful if it names every place a record lands. In a mature pipeline that is more places than it first appears: the curated table, the columnar dataset in object storage, the search index, the aggregate views, the cached API responses, the exported files sent to downstream consumers, and the raw payload in the landing zone. Each needs a store_name in the index and a documented deletion procedure.

Write the list down and test it. A quarterly exercise where someone picks a record at random, deletes it through the standard workflow, and then searches every store for traces is the cheapest possible assurance that the procedure works — and it reliably finds the store somebody forgot, which is usually the search index or a materialised view.

Backpressure Between Stages #

A crawl that fetches faster than it can write does not fail cleanly; it accumulates. Memory grows, queues lengthen, and the eventual failure is an out-of-memory kill that loses whatever was buffered. The fix is explicit backpressure: bounded queues between stages, with the upstream stage blocking rather than buffering when the downstream one falls behind.

import asyncio

class Stage:
    """One pipeline stage with a BOUNDED input queue — full means slow down."""

    def __init__(self, name: str, maxsize: int = 1000):
        self.name = name
        self.queue: asyncio.Queue = asyncio.Queue(maxsize=maxsize)

    async def submit(self, item) -> None:
        # await, never put_nowait: a full queue must propagate pressure upstream.
        await self.queue.put(item)
        metrics.queue_depth.labels(stage=self.name).set(self.queue.qsize())

The important discipline is to await on a full queue rather than dropping or expanding. Dropping loses data silently; expanding defers the failure to the memory allocator. Awaiting slows the fetch stage, which reduces the load on the target host as a side effect — a rare case where the resilient behaviour and the polite behaviour are the same behaviour.

Export queue depth per stage as a gauge and alert on sustained saturation rather than momentary spikes. A write stage that is permanently at capacity is a sizing problem; one that saturates for ten minutes each hour is usually a batching problem, and the answer is a larger batch rather than more workers. The crawl observability topic covers the panels that make this distinction visible at a glance.

Deletions Must Reach Derived Data #

The storage stage rarely has one copy of anything. A curated row feeds a materialised view, a columnar export, a search index and an aggregate table, and a delete that touches only the source leaves the record readable in four other places. Worse, several of those are rebuilt on a schedule from the source, so a partial cleanup looks correct until the next rebuild reintroduces nothing — and the stale copies simply persist untouched.

The workable arrangement is to make derived artefacts fully regenerable and cheaply rebuilt, then to treat deletion as: remove from the source, regenerate every derived artefact, verify absence. Where an artefact is too expensive to regenerate, it needs its own targeted deletion path and a place in the store list. What does not work is treating a derived store as ephemeral and therefore exempt; an export sitting in a partner’s bucket is neither ephemeral nor yours to forget about.

Keep the store list in code, next to the deletion routine, rather than in a document. A list that lives in a runbook drifts the moment someone adds a sink; a list that the deletion job iterates over fails loudly when a sink it names has disappeared, and fails a test when a sink exists that it does not name.

Verification is the step that closes it: after the regeneration, query each store for the removed key and assert nothing comes back. Automate that assertion as part of the deletion job rather than leaving it as a manual check, because the manual check is the one skipped under time pressure.

Common Compliance Pitfalls #

• Storing raw HTML forever “just in case.” Raw pages routinely contain names, emails, and session data. Wrong: an unbounded S3 prefix of scraped HTML. Right: a lifecycle rule that expires raw payloads after a defined window while curated, minimised tables persist.
• No path from a person to their rows. Wrong: personal data spread across denormalised tables with no index tying it to a subject identifier. Right: a documented lookup that an erasure workflow can execute deterministically.
• Append-only writes on a re-crawling pipeline. Wrong: INSERT every crawl and deduplicate “later.” Right: upsert on a natural key so a replay is a no-op.
• Logging the payload. Wrong: logger.info(record) dumping PII into logs that then ship to a third-party backend. Right: log the record key and metadata, never the sensitive fields.
• Retention defined but never executed. Wrong: a retention policy in a wiki and no job that enforces it. Right: a scheduled, monitored deletion sweep with a dry-run and an alert.
• Metrics without labels for compliance. Wrong: a single write counter. Right: labels distinguishing PII-bearing sinks so you can prove where personal data lives.

Frequently Asked Questions #

Should scraped data go into Postgres or a data warehouse? #

Both, usually, at different tiers. Postgres is the right home for operational, frequently-updated records where transactional upserts and constraints matter — a live product catalogue, for instance. A warehouse like BigQuery or Snowflake is better for large analytical rollups queried by analysts. Many pipelines land curated records in Postgres and periodically export snapshots to the warehouse. The structured data sinks and warehousing guide walks through the trade-offs.

How is deduplication different from idempotent writes? #

Idempotent writes solve the “same record written twice” problem at the sink using a key and an upsert. Deduplication solves the “is this even the same record?” problem — recognising that two different URLs or two crawls produced substantively identical content. You need both: upserts keep replays clean, while content hashing and near-duplicate detection keep genuinely distinct-but-redundant records out.

What retention period should I set for scraped personal data? #

Only as long as the lawful basis and purpose require, which is a legal determination rather than a technical default. Under GDPR’s storage-limitation principle you must be able to justify the period. Many teams set aggressive expiry on raw, identifiable tiers (days to weeks) and retain only anonymised aggregates longer. The mechanics of enforcing whatever period you choose are in data retention and GDPR deletion workflows.

How much observability does a small crawler actually need? #

At minimum, a request counter, an error counter, and a write counter, all exported so you can see when a crawl silently stalls or starts failing. Even a single-node scraper benefits from the Prometheus client and a basic Grafana panel described in crawl observability with Prometheus and Grafana; the cost is a few lines of instrumentation and the payoff is knowing about breakage before your dataset does.

Where does structured logging fit if I already have metrics? #

Metrics tell you that something is wrong — the error rate jumped; logs tell you why — this URL returned a challenge page and routed to quarantine. You want both, correlated by a shared ID. Emitting structured JSON events and shipping them to a searchable store is covered in structured logging and log shipping.

How long should raw responses be kept? #

Long enough to debug a parsing incident and no longer — one to two weeks covers almost every realistic case, since a breakage that has gone unnoticed for a fortnight will need a recrawl regardless. Where the responses contain personal data, the window is whatever the recorded assessment supports rather than what is operationally convenient, and it must be enforced by the same sweep job as every other retention class. A landing zone with no expiry is the single most common way a pipeline ends up holding personal data it never intended to keep.

Related guides #

• Structured Data Sinks and Warehousing — choosing Postgres, S3, Parquet, or a warehouse and writing to them idempotently.
• Deduplication Strategies for Scraped Data — exact keys, content hashing, and near-duplicate detection.
• Data Retention and GDPR Deletion Workflows — enforcing retention limits and right-to-erasure.
• Schema Validation with Pydantic — the upstream gate that produces the typed records this section stores.
• GDPR Compliance for Scraped Personal Data — the legal frame for anything personal you persist.