The discovery: counts drifting on re-run
The pipeline publishes updates as deterministic exports: same input, same payload, every time. Determinism is supposed to make replays free — run it twice, get the same result twice. But one re-run added records that were already there. Ruling entries ticked upward with no new data behind them, and the apply reported success throughout.
The unsettling part wasn't the duplication itself. It was that every layer believed its own story: the export was correct, the inserts were valid, the counts were simply… larger than before. Nobody lied; the system just had no memory of what it had already done.
The cause: re-export without a guard
Deterministic output plus plain inserts is a trap. The export faithfully reproduced every record on each run, and the loader faithfully inserted every record it was given — including the ones from last time. The pipeline was idempotent in spirit and duplicative in fact, because "don't apply twice" lived nowhere except in the operator's intentions. Intentions don't survive re-runs, retries, or 3am recovery sessions.
The decision: a ledger plus fingerprint parity
Two changes. First, an applied-payload ledger: every payload applied gets recorded, and the loader checks the ledger before touching the data — a previously applied payload becomes a safe, verifiable no-op instead of a second helping. Second, fingerprint parity around the diff-publish step, so both ends can prove they agree after every publish, not just hope they do.
The result: replays you can trust
Re-running the same payload is now the dullest event in the system: detected, skipped, logged. Recovery procedures, retries, and deliberate replays all flow through the same guard, which means the operations that used to be the most dangerous — "run it again to be sure" — are now the safest ones to reach for.
The deeper change was in how recovery feels: re-running stopped being a risk to weigh and became the default first step, because the ledger made it safe.
The lesson
Idempotency has to be recorded, not assumed. Any pipeline step that is safe to repeat only by convention will eventually be repeated in exactly the circumstances where it isn't safe. The guard catches this class, and the process catches the next one.