Crack the Sky

SPY / SPX system overview

multi-page · dark · indexed
clock America/New_York
DATA_ROOT /data/massive

This is the system design index for the running box. It exists so the code on main and Brad’s mental model stay the same document.

Rule. If these pages and the running code disagree, the function body and the tests that import it win — not a stale module docstring. Status chips mark shipped write-only 403 / not entitled not built so design intent is not mistaken for running code.

One file per section — pick a page from the index on the left or the cards below. High-level first; formula pages are stub anchors only — no derivations here. No pandas lecture, no secrets, no API keys.

The whole system on one slide

Sources into ingest into the warehouse into math into the canary SOURCES WS delayed wss://delayed.massive.com REST api.polygon.io S3 flat files files.massive.com / flatfiles IBKR Flex executions, read-only CAPTURE ingest/ jobs timers · flock · ET 15-min delayed snapshots are irreplaceable; flat file always wins WAREHOUSE /data/massive raw/ · clean/ _meta/ · logs/ parquet, dt=YYYY-MM-DD MATH marketdata/ typed as-of reads · fail-loud pricing/ own IV · BSM / CRR · SVI WATCHDOG drift_check 17:00 ET · ATM identities Healthchecks one check per scheduled job · /fail
websocket REST S3 flat file broker / on-box

Warehouse → math at a glance

1 parquet

clean/ under DATA_ROOT (default /data/massive)

2 as-of

marketdata.catalog.read_asof — last file ≤ instant, per underlying

3 own IV

Own invert of last/close — European BSM or American CRR. Never NaN; never vendor σ

4 own Greeks

SPX/SPXW analytic BSM; SPY American CRR (ATM slice on cron)

5 canary

pricing.drift_check — ATM identities first; vendor diffs optional

python -m pricing.from_market  →  python -m pricing.drift_check

Sections

Repo layout

Four packages — ingest, marketdata, pricing, signals — plus tests, deploy, scripts.
Data flow

What lands where; irreplaceable vs backfillable; write-only datasets.
Ingest

Sources, vendor constraints, every job, the NY schedule (timers + crontab fallback), session timeline, entitlements.
marketdata

Typed parquet reads, OPRA allowlist (SPY/SPX/SPXW/VIX/VIXW), as-of semantics, fail-loud validate.
pricing

Engines (European BSM / American CRR), both IV inverters, ATM curve, raw-SVI surface.
Warehouse → math

The from_market chain end to end: as-of load → own IV → own Greeks.
Daily canary

drift_check at 17:00 ET — identity-first pass/fail, and what a FAIL means.
Red day

Which Healthcheck fired, whether repair.sh can help, early close, disk/prune.
Desk conventions

Vega, theta, F, T, r — this repo’s units vs the usual vendor traps.
Not built

ClickHouse, VVIX, trading-day theta, SPY/VIX smiles — deliberately absent. SVI surface and the rates warehouse have shipped.
Target architecture

The end-to-end loop this system is aiming at — execution, ledger, and portfolio risk included. A proposal, not the box.
Latent holes

In the tree, deliberately unfixed — the desk index of IMPROVEMENTS.md.
Formula stubs

Anchors into pricing/; derivations live in code, not here.
Documented knobs

Every .env variable and code-level setting that changes behavior.
Box operations (off-repo)

Secrets, installed crontab, self-hosted runner, Healthchecks, IBKR Flex — machine state the repo cannot declare.