Engineering · Production data infrastructure
Point-in-time data at billions-of-rows scale, built to be trusted.
The strongest asset in this project is the plumbing: memory-safe, resumable, reconnect-hardened ingestion; bitemporal point-in-time contracts with automated leak tests; a self-generating coverage catalogue over the entire data universe; and a governed redaction chokepoint that this very page is published through.
Ingestion at scale — verified against source
OptionMetrics per-contract greek rows (1996–2025), 139 GB, 0-delta vs source count
TAQ opening/closing auction cross-prints, 5,772 trading days (2003–2026)
data relations catalogued → 35,021 logical table-families
point-in-time (bitemporal) data contracts under leak tests
datasets in the master inventory across 10+ vendors
Reliability engineering
Reconnect-hardened ingestion
Long multi-billion-row pulls routinely trip a research data provider's server-side connection cap. The ingestion harness is byte-bounded (RAM + disk guarded), chunks by time window, writes atomically, and reconnects on drop with backoff, resuming exactly where it stopped. The 4.31 B-row options ingest survived four mid-pull connection failures with zero row loss — the on-disk count matched the source exactly.
Point-in-time correctness
Every dataset is bitemporal: what was knowable, and when. A frozen vendor feed was revived by unioning a live source forward-only, keeping the historical panel byte-reproducible under a governed contract and a regression test — so no backtest silently gains hindsight. Leak tests fail the build if a future value can reach a past decision.
Self-maintaining coverage catalogue
A generator maps all 115,409 relations to access state, freshness, and cadence, with an honest limitations section and a doc↔data consistency test suite. When a data grant is lost, the scheduled refresh derives a skip-set from the catalogue and degrades to a visible skip, not a silent failure.
This dashboard is published through a security chokepoint
dashboard_publish.py is a single-purpose publisher: it reads disk only, never
calls the broker, never reads secrets, and refuses to run if a live-trading flag is set.
Only an explicit key whitelist can reach this public surface — broker account IDs, order
IDs, and model scores cannot leak by construction.