Crack the Sky

Red day

← overview

A Healthchecks /fail or a red box.yml coverage step is not one problem. Snapshots cannot be repaired; trades and bars can. Do not run repair.sh at a canary FAIL.

Which red

SignalWhat it isFirst move
massive-coverage-audit T-1 capture gap. Report in _meta/coverage.json. Read the table on stderr / the JSON. If it is trades/bars/WS, bash scripts/repair.sh YYYY-MM-DD.
massive-drift-check ATM identities broke (or too few ATM names, or data missing). Report in _meta/drift_check.json. Read failures[]. Math, not a hole in the warehouse — see what a FAIL means.
massive-snapshot-sweep The irreplaceable dataset missed a minute (every chain failed, or the job died). That minute is gone. Do not repair. A single chain 403 no longer fails the run; massive-snapshot-sweep-all-chains pages by absence if one chain stays down longer than the grace.
massive-ws-minute-bars / _alive Delayed AM capture died or logged zero rows. Wait for tomorrow's reconcile — the flat file overwrites WS minute bars. The liveness check is so a 09:26 death does not stay green until evening.
disk_runway FAIL Inside coverage_audit: < 60 days of snapshot growth left on the volume (WARN at 180). bash scripts/prune_raw.sh dry-run, then --apply. Never touches clean/ or raw/option_snapshots.

coverage_audit → repair.sh

bash scripts/repair.sh DATE is the standard fix when the audit reports a backfillable gap:

  1. flatfile_pull --date DATE — S3 wins, the authoritative record.
  2. reconcile --date DATE — rewrites clean option_minute_bars from that file; prior WS output is quarantined, not deleted.
  3. coverage_audit --date DATE — must go green, or the script exits 1 with snapshots cannot be repaired: they are only capturable live.

A snapshots[SPY] FAIL (< 95% of expected in-session sweeps, or no in-session files at all) is that last line. Missing snapshots_preopen / snapshots_eod is the same class: those two singletons are not in the cadence ratio, and they cannot be re-swept. A largest-gap WARN with a healthy ratio is continuity, not a hole — do not repair it.

drift_check

Re-running python -m pricing.drift_check --date DATE after a FAIL is fine (it rewrites _meta/drift_check.json). Re-running repair.sh will not change own IV. If n_stale jumped, the filter did its job — the page is skip counts, not a capture gap. If identities broke on a full ATM set, the math drifted or the as-of chain is junk; that is a code/data question, not a pull.

Disk / prune

Monthly massive-prune runs bash scripts/prune_raw.sh --apply at 03:15 on the 1st. It pings massive-prune (and the unit has OnFailure= for deaths the script cannot report). Dry-run first. Drops raw payloads older than RETAIN_DAYS (90) only when the manifest records a replacing flat file with rows kept. Never clean/, snapshots, or raw/flatfiles unless you pass --flatfiles (365-day retain, opt-in). Quarantine has its own clock (QUARANTINE_RETAIN_DAYS 30).

Early close

The audit window shrinks with market_gate (13:00 close → 13:30 end). The schedule still fires snapshot_sweep through 16:30, so those post-close files land as stray (reported, never counted) and 13:32–16:30 is unchecked. That is a known hole in latent holes, not a capture failure — do not treat ~178 stray sweeps on a 13:00 day as something repair.sh should fix.