The incident: an hour of waiting
Price data arrives wholesale: a full replacement load, tens of millions of rows, written straight into the live tables. While that write ran, reads queued behind it — nearly an hour of users waiting on a site that looked broken but was merely busy. The data was correct at the end. The experience of "getting there", not so much.
The instinct is to make the load faster. But faster-but-blocking is still blocking; it just shortens the outage you scheduled for yourself. The problem was never speed. It was that readers and writers were sharing the same table at the same time.
The pattern: build beside production, then swap
So the load moved beside production instead of through it. Each update builds into a shadow table — full write, no readers waiting, all the time it needs — and only when the new data is complete does the swap happen: an atomic rename that exchanges old for new in a single step. Readers see the old data or the new data, never a half-written middle. The outage window shrank from an hour to nothing observable.
The safety: preflight before the swap, not after
Atomicity alone isn't trust; a fast swap of bad data is just a faster mistake. So the shadow gets verified before it goes live: schema checks, datadir awareness, guards against swapping into a broken state. Verification happens before the swap, not after — the point of no return sits behind every check that matters.
The result: nothing — and that's the point
From a user's perspective, a price update now looks like nothing at all. Numbers refresh between one page load and the next, with no waiting, no degradation, no maintenance window. The most successful deploy pattern is the one nobody notices.
The lesson
The deploy pattern matters as much as the data. Don't claim zero risk — claim bounded, verified risk. Readers never blocked, bad snapshots never promoted, and a swap so small there's almost nothing left in it to fail.