Crack the Sky

Data flow

What lands where

Capture lands parquet under DATA_ROOT. Math never reads the vendor’s IV/greeks as inputs. The canary re-runs the same chain every weekday at 17:00 ET.

1 parquet

clean/ under DATA_ROOT (default /data/massive)

2 as-of

marketdata.catalog.read_asof — last file ≤ instant, per underlying

3 own IV

Own invert of last/close — European BSM or American CRR. Never NaN; never vendor σ

4 own Greeks

SPX/SPXW analytic BSM; SPY American CRR (ATM slice on cron)

5 canary

pricing.drift_check — ATM identities first; vendor diffs optional

python -m pricing.from_market → python -m pricing.drift_check

Irreplaceable vs backfillable

Irreplaceable: option_snapshots (+ derived forwards). No historical snapshot endpoint. Sweep every minute.
Backfillable: trades, minute bars, day bars from S3. Flat file always wins at reconcile.

Write-only and derived

Decision history: decision_log — the append-only decision record: one row per entry / exit / roll / no-trade event carrying inputs, signal values, gate outcome, rationale, and versions. No producer yet; the backtester writes first. A second write is a second file, never a quarantine, and reads take the whole partition (ingest.common.decision_log.read_decisions) — as-of last-file-wins would hide earlier events, so read_asof refuses the dataset.
Write-only today: contracts_expired, ibkr_executions. Captured; nothing in pricing/marketdata/signals consumes them yet.
Now consumed: forwards is no longer write-only — from_market joins per-expiry parity F. underlying_day_bars and dividends feed signals.spot.
Derived: atm_term_structure — one row per (date, root, expiry), built from option_day_bars by pricing.term_structure at 12:00 ET. Not captured: the whole archive rebuilds in ~7 minutes via scripts/build_term_structure.py, so it is safe to delete and safe to recompute after any change to the reduction. Built from day bars rather than snapshots precisely because day bars reach back to 2022-08-31 while snapshots only begin in 2026. vol_surface is the same shape one dimension up: one raw-SVI smile fit per (date, root, expiry) off the same day bars, scheduled at 12:15 ET, rebuilt by scripts/build_surface.py, read back by load_surface. spy_spot is one row per session: SPY close from underlying_day_bars where it exists, else S = F e^{-(r-q)T} from the shortest-DTE SPY term-structure row and the dividend stream; scheduled 12:05 ET, rebuilt by scripts/build_spy_spot.py --force, which also writes the overlap calibration to _meta/spy_spot_calibration.json. rv_forecast is one row per (origin session, horizon): a walk-forward HAR-RV fit on spy_spot at 3/5/10/21/32 sessions (~4–45 calendar days, the 5–45 DTE book), carrying the lognormal forecast distribution (log_rv_mean/log_rv_sd) and an 80% annualised-vol band rather than a point estimate; run by python -m signals.har_rv (T-1, currently by hand) and rebuilt over the archive by scripts/build_rv_forecast.py. Point-in-time by construction: the fit at an origin trains only on targets that ended at or before it.