Validating Scraped Dates and Currency Values with Pydantic v2 #

Scraped date strings and price fields are where type discipline usually breaks down: a listing page emits "Jul 3, 2026", "03/07/2026", and "hace 2 días" on the same crawl, and a product feed prints "€1.299,00" next to "$1,299.00". This page is a focused recipe for coercing those messy values into datetime and Decimal fields safely, using Pydantic v2 field_validator hooks. It sits under Schema Validation with Pydantic within the broader Data Parsing & Transformation Pipelines section, and assumes you already validate the surrounding record structure — here we drill into the two field types that cause the most silent corruption.

Problem Framing #

Dates and money share a dangerous property: the wrong parse still looks plausible. 03/07/2026 is 3 July in most of the world and 7 March in the US; nothing in the string tells you which, so a naive parser will happily produce a valid-but-wrong datetime. Money is worse, because floating-point arithmetic silently loses precision — 0.1 + 0.2 != 0.3 — and thousands separators differ by locale, so 1.299 is either “one thousand two hundred ninety-nine” (German) or “one point two nine nine” (US). If either error reaches your warehouse, downstream aggregates, tax calculations, and time-window queries are quietly wrong and nobody sees an exception.

None of these can be resolved from the string alone — the site tells you.

The fix is to make the parsing rules explicit at the validation boundary, store money as Decimal, store timestamps as timezone-aware UTC, and reject anything the rules cannot resolve rather than guessing.

Step-by-Step Implementation #

Model money as Decimal, never float. Annotate the field as Decimal and use a field_validator(mode="before") to strip currency symbols and normalise separators before Pydantic coerces the value.
Detect the decimal convention from the string shape rather than assuming the site’s locale — the last . or , that is followed by exactly two digits is the decimal separator; everything else is a grouping separator to delete.
Parse dates with dateutil but pass dayfirst/yearfirst explicitly per source, so ambiguous dd/mm vs mm/dd strings resolve deterministically instead of by library default.
Normalise every timestamp to UTC. Attach the source site’s timezone when the string is naive, then convert with astimezone(timezone.utc) so all stored times are comparable.
Fail loudly on unparseable input by raising ValueError inside the validator — Pydantic wraps it in a ValidationError your dead-letter routing already handles.

Storing minor units as integers removes floating-point drift entirely.

# money_and_dates.py
from decimal import Decimal, InvalidOperation
from datetime import datetime, timezone
from zoneinfo import ZoneInfo
import re

from dateutil import parser as dateutil_parser
from pydantic import BaseModel, field_validator, Field, ValidationInfo

_CURRENCY_SYMBOLS = "$€£¥₹"
# Decimal separator = a '.' or ',' followed by exactly 2 digits at end of number.
_DECIMAL_TAIL = re.compile(r"[.,](\d{2})\b")


def _to_decimal(raw: str) -> Decimal:
    s = raw.strip()
    for sym in _CURRENCY_SYMBOLS:
        s = s.replace(sym, "")
    s = re.sub(r"[A-Za-z\s ]", "", s)  # drop 'USD', NBSP, stray letters
    if not s:
        raise ValueError("empty money string")
    # Identify the decimal separator by the two-digit tail; delete all others.
    m = _DECIMAL_TAIL.search(s)
    if m:
        dec_sep = s[m.start()]
        s = s.replace("." if dec_sep == "," else ",", "")  # kill grouping
        s = s.replace(dec_sep, ".")
    else:
        s = s.replace(",", "").replace(".", "")  # integer amount, no decimals
    try:
        return Decimal(s)
    except InvalidOperation as exc:
        raise ValueError(f"unparseable money value: {raw!r}") from exc


class ScrapedListing(BaseModel):
    price: Decimal = Field(gt=0, description="Money as Decimal, never float")
    currency: str = Field(pattern=r"^[A-Z]{3}$")  # ISO 4217
    published_at: datetime  # stored timezone-aware, UTC
    source_tz: str = Field(default="UTC", exclude=True)  # e.g. "America/New_York"
    dayfirst: bool = Field(default=False, exclude=True)  # per-source hint

    @field_validator("price", mode="before")
    @classmethod
    def parse_price(cls, v: object) -> object:
        return _to_decimal(v) if isinstance(v, str) else v

    @field_validator("published_at", mode="before")
    @classmethod
    def parse_published_at(cls, v: object, info: ValidationInfo) -> object:
        if not isinstance(v, str):
            return v
        try:
            dt = dateutil_parser.parse(v, dayfirst=info.data.get("dayfirst", False))
        except (ValueError, OverflowError) as exc:
            raise ValueError(f"unparseable date: {v!r}") from exc
        if dt.tzinfo is None:  # naive -> attach source tz, then normalise to UTC
            tz = ZoneInfo(info.data.get("source_tz", "UTC"))
            dt = dt.replace(tzinfo=tz)
        return dt.astimezone(timezone.utc)

Because field_validator runs in field-declaration order, source_tz and dayfirst are populated before published_at and are available on info.data. Keep those hint fields per source rather than global, and store them alongside your selector config so a US site and a German site can share the same model class with different parsing behaviour.

Verification & Testing #

Assert on concrete ambiguous inputs — this is the only reliable way to prove the locale logic holds. Test both separator conventions and both date orderings.

Each case has a wrong answer that still parses without raising.

# test_money_and_dates.py
from decimal import Decimal
from datetime import timezone
import pytest
from money_and_dates import ScrapedListing, _to_decimal


@pytest.mark.parametrize("raw,expected", [
    ("$1,299.00", Decimal("1299.00")),   # US grouping
    ("€1.299,00", Decimal("1299.00")),   # German grouping
    ("£49.99", Decimal("49.99")),
    ("1 234,50 USD", Decimal("1234.50")),  # NBSP grouping
])
def test_money_locale(raw, expected):
    assert _to_decimal(raw) == expected


def test_date_dayfirst_disambiguation():
    us = ScrapedListing(price="1", currency="USD",
                        published_at="03/07/2026", dayfirst=False)
    eu = ScrapedListing(price="1", currency="EUR",
                        published_at="03/07/2026", dayfirst=True)
    assert us.published_at.month == 3   # March
    assert eu.published_at.month == 7   # July
    assert us.published_at.tzinfo == timezone.utc


def test_unparseable_rejected():
    with pytest.raises(Exception):
        ScrapedListing(price="free", currency="USD", published_at="2026-01-01")

Run pytest -q in CI as a gate before deploying any selector change. Because Decimal("1299.00") == Decimal("1299.0"), compare with the exact scale you expect, or normalise with .quantize() when the scale itself matters for storage.

Compliance & Operational Guardrails #

Store money as Decimal end to end — write it to a NUMERIC/DECIMAL column, not FLOAT. If a price feeds billing, invoicing, or tax reporting, float rounding is an auditable financial error, not a cosmetic one.
Record the parse decision, not just the result. Log the raw string, the resolved dayfirst, and the applied source_tz alongside the record so data lineage stays defensible; this pairs with the audit-log patterns in the parent Schema Validation with Pydantic guide.
Timestamps that carry personal data (account activity, review dates) fall under data-minimisation rules — normalise them to UTC but do not enrich beyond what the source published, and hash or drop any co-located PII at the same validation boundary before persistence.
Reject rather than guess on ambiguous input. A quarantined record in a dead-letter queue is recoverable; a wrong-but-valid date silently poisoning a time-series is not.

Resolving Ambiguity From the Site, Not the String #

03/04/2026 cannot be resolved by looking at it. The information that disambiguates it lives on the page or on the host: an hreflang attribute, an html lang, a currency code beside the price, a time zone in an embedded structured-data block, or simply the site’s country of operation. Capture that context during extraction and pass it into the validator rather than letting each parser guess.

from dataclasses import dataclass
from datetime import datetime
from zoneinfo import ZoneInfo

@dataclass(frozen=True)
class SiteLocale:
    """Parsing context resolved per host, not per value."""
    date_order: str        # "DMY" | "MDY" | "YMD"
    decimal_sep: str       # "." | ","
    default_currency: str  # ISO 4217
    timezone: str          # IANA name

ORDER_FORMATS = {
    "DMY": ("%d/%m/%Y", "%d-%m-%Y", "%d.%m.%Y"),
    "MDY": ("%m/%d/%Y", "%m-%d-%Y"),
    "YMD": ("%Y-%m-%d", "%Y/%m/%d"),
}

def parse_date(raw: str, locale: SiteLocale) -> datetime:
    text = raw.strip()
    for fmt in ORDER_FORMATS[locale.date_order]:
        try:
            naive = datetime.strptime(text, fmt)
        except ValueError:
            continue
        return naive.replace(tzinfo=ZoneInfo(locale.timezone))
    raise ValueError(f"unparseable date {raw!r} for order {locale.date_order}")

Attaching the site’s time zone rather than assuming UTC matters for anything time-sensitive: a listing timestamped 23:30 on a site operating nine hours from UTC lands on the wrong calendar day in every downstream report if the zone is dropped. Store the value as an aware timestamp, convert to UTC at the sink, and keep the original zone as a column so the local rendering can be reconstructed.

Never Store Money as a Float #

0.1 + 0.2 is not 0.3 in binary floating point, and a pipeline that aggregates a million scraped prices as floats accumulates a visible error. Parse to Decimal, store integer minor units with an explicit ISO 4217 code, and do arithmetic on integers.

from decimal import Decimal, ROUND_HALF_UP

EXPONENTS = {"JPY": 0, "KRW": 0, "USD": 2, "EUR": 2, "GBP": 2, "BHD": 3, "KWD": 3}

def to_minor_units(amount: Decimal, currency: str) -> int:
    exponent = EXPONENTS.get(currency)
    if exponent is None:
        raise ValueError(f"unknown currency exponent for {currency!r}")
    quantum = Decimal(1).scaleb(-exponent)
    return int((amount / quantum).quantize(Decimal(1), rounding=ROUND_HALF_UP))

The exponent table is the detail most implementations omit. Assuming two decimal places turns ¥1,200 into 120,000 minor units — a hundredfold error that passes every type check and every range constraint, and which is only discovered when someone compares a total against the source. Currencies with zero and three decimal places are common enough that the assumption will eventually be wrong.

Symbols are ambiguous and should never be the source of truth for the currency: $, kr, and £ each map to several currencies. Prefer an explicit code from structured data or the site’s locale record, and quarantine rather than guess when neither is available — a wrong currency is far more damaging downstream than a missing record, and the quarantine path exists precisely for this case.

Common Mistakes #

Using float for money. It looks fine in tests with round numbers and corrupts sums at scale. Always Decimal, and strip separators before construction — Decimal("1,299") raises InvalidOperation.
Trusting dateutil’s default dayfirst=False for European sources. The default silently turns 3 July into 7 March. Set the flag per source and test both orderings.
Storing naive datetimes. A timestamp without a timezone is only meaningful next to the machine that made it. Attach the source zone, convert to UTC, and store aware datetime objects.

Frequently Asked Questions #

Why not use Pydantic’s built-in `condecimal` or `AwareDatetime` types instead of custom validators? #

Those constrain the value after it is already a Decimal or datetime, but they cannot clean a "€1.299,00" string or disambiguate 03/07/2026 — coercion fails before the constraint runs. Use a mode="before" validator to normalise the raw string first, then layer AwareDatetime or a condecimal(gt=0) constraint on top for the final guard.

How do I handle relative dates like “2 days ago” or “yesterday”? #

dateutil cannot parse those. Add a small pre-processing branch in the before validator that matches known relative phrases per language and subtracts a timedelta from a captured crawl timestamp. Always anchor to the crawl time, not datetime.now() at validation time, so re-processing an old batch yields the same absolute date.

Should currency conversion happen inside the validator? #

No. Validation should record the amount and its original ISO 4217 currency exactly as published. FX conversion is a separate transformation stage with its own dated rate table, and folding it into validation destroys the audit trail linking the stored value to the source page.

Schema Validation with Pydantic — the parent technique: models, strict mode, and dead-letter routing.
Normalizing Nested JSON Responses — flatten and reshape payloads before these field validators run.
Upserting scraped records into Postgres — persist Decimal and UTC timestamptz values without precision loss.