Deduplicating records with content hashing #

The same entity reaches your pipeline through many doors: two URLs that render identical product data, a re-crawl that returns unchanged content, a paginated feed that overlaps the previous page. Storing all of them wastes space and skews analytics. Content hashing gives each record a compact fingerprint derived from its meaningful fields, so “have I already stored this exact content?” becomes a single index lookup. This page explains how to canonicalise a record, compute a stable SHA-256 over its normalised fields, store the digest, and skip on seen. It is a detail page under Deduplication Strategies for Scraped Data within the Pipeline Storage, Deduplication & Monitoring section.

Problem Framing #

There are two distinct dedup questions and content hashing answers one of them precisely. Entity deduplication asks “are these two records the same thing?” — that is a natural-key or fuzzy-matching problem. Content deduplication asks “is this record byte-for-byte the same content I already have?” — that is a hashing problem. Content hashing is exact, cheap, and unambiguous: identical meaningful fields produce an identical digest, and any change produces a completely different one.

Hash the record you keep, not the bytes you happened to receive.

The trap is stability. A hash is only useful if the same logical record always produces the same digest. Serialise a Python dict without sorting keys and the hash flips between runs because dict ordering, whitespace, and float formatting all vary. Include a scraped_at timestamp in the hash and every re-crawl looks “new”. The entire technique lives or dies on a disciplined canonicalisation step that strips volatile fields and produces one deterministic byte string per logical record.

Step-by-Step Implementation #

Select the semantic fields. Decide which fields define the content. Exclude volatile metadata — fetch timestamp, crawl id, proxy IP, response latency — that changes without the content changing.
Normalise each field. Trim and collapse whitespace, lowercase where case is not significant, normalise Unicode to NFC, and format numbers and dates to a fixed representation. This is what makes " 9.90 " and "9.9" hash the same when they mean the same price.
Serialise deterministically. Dump to JSON with sort_keys=True, no insignificant whitespace, and ensure_ascii=False, then encode as UTF-8. Sorted keys remove dict-ordering nondeterminism.
Hash with SHA-256 over the canonical bytes to get a stable hex digest.
Store and skip on seen. Keep the digest in a unique index (or a Redis set for hot lookups) and drop records whose hash already exists.

Fixed field order is what makes the hash reproducible across releases.

import hashlib
import json
import unicodedata

# Fields that must NOT enter the hash: they change without the content changing.
VOLATILE = {"scraped_at", "crawl_id", "proxy_ip", "response_ms", "etag"}

def _normalise_scalar(value):
    if isinstance(value, str):
        # NFC-normalise, strip, and collapse internal whitespace runs.
        text = unicodedata.normalize("NFC", value).strip()
        return " ".join(text.split())
    if isinstance(value, float):
        # Fixed formatting so 9.9 and 9.90 do not diverge.
        return format(value, ".4f")
    return value

def canonicalise(record: dict) -> bytes:
    """Produce one deterministic byte string per logical record."""
    cleaned = {
        k: _normalise_scalar(v)
        for k, v in record.items()
        if k not in VOLATILE and v is not None
    }
    return json.dumps(
        cleaned, sort_keys=True, separators=(",", ":"), ensure_ascii=False
    ).encode("utf-8")

def content_hash(record: dict) -> str:
    """Stable SHA-256 hex digest over the record's normalised fields."""
    return hashlib.sha256(canonicalise(record)).hexdigest()

Wire the skip-on-seen check into the write path. A database unique index on the digest is the durable source of truth; a Redis set in front of it keeps the common “already seen” case off the database:

def store_if_new(conn, redis, record: dict) -> bool:
    digest = content_hash(record)
    # Fast path: SADD returns 0 if the member already existed.
    if redis.sadd("seen:content", digest) == 0:
        return False  # duplicate — skip silently
    with conn.cursor() as cur:
        # Durable guard: the unique index is authoritative if Redis is cold.
        cur.execute(
            "INSERT INTO record (content_hash, payload) VALUES (%s, %s) "
            "ON CONFLICT (content_hash) DO NOTHING RETURNING id",
            (digest, json.dumps(record)),
        )
        inserted = cur.fetchone() is not None
    conn.commit()
    return inserted

The Redis SADD is an optimisation, not the guarantee — the ON CONFLICT (content_hash) DO NOTHING on a UNIQUE index is what keeps the store correct even if the cache is flushed or a second worker races. This same digest feeds the conditional updates described in upserting scraped records into Postgres, so compute it once and pass it downstream.

Verification & Testing #

Assert the two properties that make the hash trustworthy — determinism and volatility-insensitivity:

Without the version column, changing the rule silently invalidates history.

def test_hash_is_stable_and_ignores_volatile_fields():
    a = {"sku": "A1", "title": "Widget ", "price": 9.9,
         "scraped_at": "2026-07-05T10:00:00Z", "proxy_ip": "10.0.0.1"}
    b = {"price": 9.90, "sku": "A1", "title": "Widget",   # reordered + reformatted
         "scraped_at": "2026-07-05T11:30:00Z", "proxy_ip": "10.0.0.9"}
    # Same content, different volatile metadata and key order → same digest.
    assert content_hash(a) == content_hash(b)

    c = {**a, "price": 10.0}                                # real change
    assert content_hash(a) != content_hash(c)

Confirm the collision surface is negligible with a git-style sanity check: SHA-256 over normalised fields makes an accidental collision astronomically unlikely, so any two records sharing a digest can be treated as identical content. Measure the dedup rate in production by comparing SADD hits to total records — a sudden drop toward zero usually means a volatile field leaked into canonicalise.

Compliance & Operational Guardrails #

A hash of personal data is still personal data. A SHA-256 digest over a name and address is a pseudonym, not anonymisation — it is reversible by anyone who can re-hash a guess. Keep hashed personal fields inside the same retention and access controls as the raw record, per GDPR compliance for scraped personal data.
Version your canonicalisation logic. If you change which fields or normalisation rules feed the hash, store a hash_version alongside the digest so old and new digests are never compared as equals.
Make dedup auditable. Log the digest and the skip decision so an audit can show why a record was or was not stored, without retaining the duplicate payload itself.

A Canonical Serialisation Is the Whole Game #

A content hash is only reproducible if the bytes fed into it are reproducible. Python dictionaries preserve insertion order, floats have several textual representations, and None versus an empty string is a distinction the extractor may not make consistently. Any of these makes the same logical record hash differently between two runs, which silently converts deduplication into duplication.

import hashlib
import json
import unicodedata
from decimal import Decimal

HASH_VERSION = "content/2"

IDENTITY_FIELDS = ("title", "price_minor", "currency", "seller_id", "description")

def _canonical(value):
    if value is None:
        return None
    if isinstance(value, str):
        # NFC so composed and decomposed accents hash identically; whitespace collapsed.
        return " ".join(unicodedata.normalize("NFC", value).split())
    if isinstance(value, Decimal):
        return str(value.normalize())
    if isinstance(value, float):
        raise TypeError("floats are not hashable reproducibly; use Decimal or minor units")
    return value

def content_hash(record: dict) -> tuple[str, str]:
    payload = {field: _canonical(record.get(field)) for field in IDENTITY_FIELDS}
    encoded = json.dumps(payload, sort_keys=True, ensure_ascii=False,
                         separators=(",", ":")).encode("utf-8")
    return hashlib.sha256(encoded).hexdigest(), HASH_VERSION

Four decisions make this stable. A fixed field tuple means adding a field to the record does not change existing hashes. sort_keys=True removes dictionary ordering from the equation. Unicode normalisation collapses the two ways of encoding an accented character, which differ byte-wise and look identical. And rejecting floats outright prevents a value that round-trips differently between platforms from entering the hash at all.

HASH_VERSION is stored beside the hash on every record. When the field tuple or the normalisation changes, the version changes, and rows with the old version are recognisably computed under different rules rather than appearing to be genuinely different records.

Rehashing After a Rule Change #

Changing the hash rule invalidates every stored hash, and the migration needs to be deliberate. The safe sequence: add the new version alongside the old rather than replacing it; backfill the new hash for existing rows in batches; verify that the collapse rate under the new rule is close to what was expected; and only then switch the deduplication query to the new column. Keeping both for one full crawl cycle gives a direct comparison of how many records each rule considers distinct — which is the only reliable way to discover that a “small” normalisation change merged two thousand records that should have stayed separate.

Common Mistakes #

Hashing the raw serialised dict. Unsorted keys, stray whitespace, and float formatting make the digest unstable across runs, so nothing ever matches. Canonicalise first.
Including volatile fields. A scraped_at or proxy_ip inside the hash makes every re-crawl look new, defeating the entire purpose.
Trusting the cache as the source of truth. A Redis set alone loses all history on a flush; back it with a UNIQUE index and ON CONFLICT DO NOTHING.

Frequently Asked Questions #

Why SHA-256 rather than a faster hash like MD5 or a CRC? #

CRCs and short hashes have high collision rates at scale — with millions of records you will get accidental collisions that silently drop distinct content. SHA-256’s 256-bit space makes accidental collisions effectively impossible, and the CPU cost is trivial next to network and parsing time. Non-cryptographic hashes such as xxHash are acceptable only for a first-pass filter that is confirmed by an exact comparison.

How do I dedup near-duplicates that differ by a word or a timestamp in the body? #

Content hashing is exact by design and will not catch near-duplicates — that is a feature, because it never merges records that genuinely differ. For fuzzy matching (say, the same article republished with minor edits), use a similarity technique such as MinHash or SimHash as a separate stage; keep exact content hashing as the fast, unambiguous first line of defence.

Deduplication Strategies for Scraped Data — the parent technique comparing exact, fuzzy, and key-based dedup.
Upserting scraped records into Postgres — use the content hash to drive conditional, idempotent writes.
Deduplicating URLs with a Redis seen-set — the fetch-time analogue that stops you re-downloading the same URL.