FIELD NOTES

Real work. Documented.

Technical notes from a production data platform: what was built, what broke, what was measured, and what changed as a result.

FEATURED CASE STUDY

MTG Scout: Building a Collection Data Platform

MTG Scout is a collection management platform for Magic: The Gathering, tracking everything from individual cards and sealed products to prices, spreadsheet imports, and Commander decks. Built on Django and MariaDB, the platform combines a daily data pipeline, S3-backed image storage, automated production deployments with Ansible, and first-party operational telemetry.

The field notes below document the real-world decisions, implementation work, and ongoing development behind the platform.

LATEST

Notes from the build.

12,408 changes

INSERT3,120

UPDATE9,210

DELETE78

APPLY 163s · BATCH 500 · RESUMABLE
UPDATE SHARE 74%

DATA PIPELINE

Chunked applies beat database lock timeouts

Large card-data syncs were failing when long-running transactions hit database lock timeouts. The solution was to apply changes in small, individually committed batches, coordinated by a staging lock. If a run is interrupted, processing can resume rather than restarting from scratch.

The lesson: Design data pipelines to handle interruption and recovery from day one.

CONVERGED

LAG 7d 0–3d: █▅▅▂▂▁▁

LAG 0d · DELAY 0d
CONVERGE 41s · PLANE uncompressed

DATA QUALITY

The CDN was serving yesterday's data

Morning syncs repeatedly ingested stale upstream dumps because the CDN continued serving a cached archive after newer data was available. Switching ingestion to the uncompressed data source eliminated this particular source of staleness.

The lesson: Verify what the cache is actually serving before blaming the underlying data.

2 replays skipped

APPLIED 31 · SKIPPED 2
PARITY PASS · DUPLICATES 0.00%

DATA RELIABILITY

A replay guard for silent duplicates

A deterministic re-export quietly duplicated ruling records. The fix was an applied-payload ledger that records processed payloads, making replays safe and duplicate processing detectable.

The lesson: Idempotency must be recorded, not assumed.

237,952 objects verified

CHECKSUMS 200/200 · ANON READ PASS
DERIVATIONS 3 · IMMUTABLE URLS

INFRASTRUCTURE

237,952 images moved to object storage

The full card-image library (~150 GB) outgrew the application host. The migration to S3-compatible object storage was verified through object counts, total bytes, checksums, and anonymous-read tests. The resulting setup uses immutable, deterministic URLs and three image-size derivations.

The lesson: Bulk migrations earn trust through verification, not hope.

top actions · 24h

search12,400

card-view8,300

deck-view1,900

22,600 EVENTS · p50 28–55ms
ROLLED UP HOURLY · NO VISITOR DATA

OPERATIONS

Telemetry that shows what the system costs

ActionStats instruments application actions, recording what ran, how long it took, and the associated resource cost — separately from visitor analytics. These operational measurements help prioritize pipeline improvements and infrastructure work.

The lesson: Measure the cost of running the system, not just the traffic it receives.

14,380,000 rows swapped

BEFORE~3600s

NOW0s

SWAP ATOMIC · PREFLIGHT PASS

INFRASTRUCTURE

Price updates without taking the site offline

Wholesale price loads blocked reads for nearly an hour. The solution was to load the data into a shadow table, run preflight checks on disk space and schema, then swap tables atomically.

The lesson: How data is deployed matters as much as the data itself.