Ingest
Three pipes into one disk tree. Everything live is 15-minute delayed.
The schedule lives in deploy/schedule.json and is rendered
into systemd user timers; the crontab stays installed through the
cutover. Every line goes through
scripts/cronjob.sh, which takes the job's
flock and — when a previous run still holds it — logs a
job_skipped event instead of dropping the run in silence.
Shared runner is ingest.common.cli.run_job (settings,
market gate, JSONL logs, per-job Healthchecks /start and
/fail).
Outbound REST rate is bounded per box, not per process: the
token bucket keeps its tokens in
_meta/ratelimit.json under flock, and every
capture job except snapshot_sweep draws at low priority
and stops at a reserve. Snapshots cannot be re-swept; trades and bars
come back from the flat files, so they are the ones that yield.
Massive / Polygon constraints that affect design
- One delayed-options websocket slot. Wildcards
AM.O:SPY*ACK and deliver nothing; subscribe explicitAM.O:<ticker>in chunks of 3,000. - No
X-RateLimit-*headers; 429s happen under burst, not under sustained pacing (probed to 100 rps clean, 2026-09-03). Shared cross-process token bucket: sustainedMASSIVE_MAX_RPSdefault 60,MASSIVE_BURSTheld at 40, plus retries. - Auth is
?apiKey=; paginationnext_urlomits the key — the client re-appends it. - SPXW weeklies list under
underlying_ticker=SPX/ snapshot pathI:SPX. Querying SPXW as an underlying returns 0 rows. - Same-day SPY equity minute aggs 403 →
underlying_barsis T-1. Intraday SPY lives on the option snapshot’sunderlying_price. - I:SPX / I:VIX index levels 403 everywhere. SPX spot is recovered from put-call parity on the chain →
clean/forwards. - Option NBBO REST and
quotes_v1flat file are 403. No bid/ask, no mid. Market price is last trade else day close. - Timestamps stored exactly as delivered (ns/ms epoch, UTC). Never converted in ingest.
Jobs that exist
Scheduled jobs plus one deliberate non-cron tool. Cadence is Eastern
time from deploy/schedule.json — the one canonical
schedule, rendered into systemd user timers by
deploy/ansible/playbook.yml.
| Job | Cadence (ET) | Days | What it writes |
|---|---|---|---|
| contracts_sync rest |
08:00 and 16:30 | Mon–Fri | Option contract universe for SPY, SPX (SPXW listed under SPX), and VIX (VIXW listed under VIX). Raw JSONL + clean contracts parquet. |
| dividends_sync rest |
08:00 | Mon–Fri | SPY dividends and splits. Clean dividends / splits. |
| ws_minute_bars ws |
starts 09:25, holds until ~16:35 | Mon–Fri | Delayed option minute bars (AM events). Hourly JSONL under raw/option_minute_bars_ws. Logs ws_stats every 5 min. Zero rows at window end fails the Healthcheck. |
| snapshot_sweep rest |
every 1 min, 09:30–16:30 · 09:05 pre-open | Mon–Fri | Full-chain snapshots for SPY, I:SPX, and VIX, concurrent. Clean option_snapshots (IV/greeks/OI populated on this tier) and clean/forwards (parity, discounted when the Treasury curve is present). Raw JSONL off unless --raw. |
| trades_watchlist rest |
every 5 min, 09:30–16:30 | Mon–Fri | Incremental trades, 7–45 DTE / ±15% moneyness, per-underlying reference (SPX is not banded on SPY). Cursor in _meta/trades_cursor.json. |
| snapshot_sweep --eod rest |
16:35 | Mon–Fri | Same snapshot job, tagged EOD in filenames and logs. |
| pricing.drift_check | 17:00 | Mon–Fri | Identity canary. See Daily canary. Not an ingest job; shares Healthchecks slug pattern massive-drift-check. |
| underlying_bars --prev-trading-day rest |
08:05 | Tue–Sat | SPY 1-minute equity bars for the previous session only. Clean underlying_minute_bars. |
| grouped_daily rest |
08:10 · default date = T-1 | Tue–Sat | Whole-market grouped daily; keeps SPY plus VIX ETF proxies (VIXY/UVXY/VXX). Clean underlying_day_bars write-only. |
| backfill_underlying | 02:30 | Daily | Oldest-first walk of the rolling ~2-year equity-aggregate window (underlying_bars + grouped_daily in-process). 403 at the old edge is expected; a real failure is not. scripts/backfill_underlying.py --max-sessions 40 --sleep 6. |
| flatfile_pull s3 |
11:05 · default date = T-1 | Tue–Sat | OPRA trades / minute aggs / day aggs csv.gz, filtered to roots SPY / SPX / SPXW / VIX / VIXW (not SPYL… / VIXY…). Manifest _meta/flatfile_manifest.json. |
| reconcile s3 |
11:30 · default date = T-1 | Tue–Sat | WS vs flat-file minute bars; rewrites clean option_minute_bars from the flat file. Flat file always wins. |
| term_structure derived |
12:00 · default date = T-1 | Tue–Sat | Reduces option_day_bars to atm_term_structure: one row per (root, expiry) with the parity forward, the ATM strike and both leg IVs. Lives in pricing/, not ingest/ — it needs numpy/scipy. Derived, so a missed day is rebuilt with scripts/build_term_structure.py; scheduled anyway because a reduction that stops producing is as invisible as a capture that stops. |
| spy_spot derived |
12:05 · default date = T-1 | Tue–Sat | One SPY cash level per session into spy_spot. Prefers underlying_day_bars close (src=bars); otherwise inverts the shortest-DTE SPY parity forward with q from resolve_q and the dividends stream (src=parity). resid is proxy minus actual on overlap. Lives in signals.spot. Rebuild + overlap calibration: scripts/build_spy_spot.py --force (without --force, already-written parity rows are skipped). |
| surface derived |
12:15 · default date = T-1 | Tue–Sat | Fits raw SVI per (root, expiry) from the same option_day_bars as the ATM curve into vol_surface. Lives in pricing/. Surface.from_rows / load_surface read the landed parameters back so consumers do not refit. Derived, so a missed day is rebuilt with scripts/build_surface.py; scheduled anyway because a reduction that stops producing is as invisible as a capture that stops. After term_structure, before coverage_audit. |
| coverage_audit | 12:30 · default date = T-1 | Tue–Sat | Reports on the absence of a job: sweep counts vs market_gate schedule, largest gap, manifest, non-empty partitions, T-1 vol_surface freshness (both SPX and SPXW), WS output, per-root ticker counts, and disk runway. Cadence is measured over the continuous 09:30–16:30 window only: the scheduled 09:05 pre-open and 16:35 EOD sweeps are asserted separately (snapshots_preopen / snapshots_eod), and sweeps run by hand outside the session are reported but never counted — folding all three together made SPY and SPX WARN every single day on a healthy box, and let out-of-hours runs pad the ratio over a real hole. Exit 1 → Healthchecks /fail. Looks at T-1 only — see history_audit for the archive. |
| history_audit | 13:00 · whole archive | Saturday | Walks every weekday since 2022-08-31 and asserts the three flat-file datasets are present. Tells a holiday from a hole by asking the vendor whether a flat file exists for that date — _meta/holidays.json only covers upcoming dates, and a hardcoded list would miss ad-hoc closures. Answers cache into _meta/trading_days.json, the verified historical calendar. A date with data on disk is never probed, so a healthy run makes no network calls (~4s for 1,044 days). Exit 1 → /fail. |
| holidays_sync rest |
07:00 | Sunday | Rewrites _meta/holidays.json from /v1/marketstatus/upcoming. Code injects --force so the weekday gate does not skip Sunday. |
| contracts_sync --expired rest |
09:00 | Saturday | Expired-contract pass into contracts_expired write-only. Code injects --force so the trading-day gate does not skip Saturday — without it this job exited 0 before doing anything and still pinged Healthchecks ok, so it monitored as healthy having never written a partition. The injected force un-gates the expired pass only: the live contracts pass re-checks the calendar itself and is skipped (contracts_live_pass_skipped), so no session-dated partition is written for a day with no session. |
| prune_raw.sh --apply | 03:15 on the 1st | Monthly | Drops raw payloads older than RETAIN_DAYS (90) for rebuildable datasets. Never touches clean/, snapshots, flatfiles, or reference. Pings massive-prune from cronjob.sh (bash, not ingest.jobs). |
| rates_sync rest |
08:20 | Tue–Sat | Treasury par yield curve + inflation series via Massive /fed/v1 (own budget, 0.2 rps — only a --full rebuild pages). Clean treasury_yields / inflation. Daily run is 2 requests; history already walked back to 1962. |
| ibkr_executions | 18:30 | Mon–Fri | Broker fills via the IBKR Flex Web Service (read-only reporting, no gateway). Clean ibkr_executions; a day with no fills exits cleanly. Token expires ~yearly — see box operations. |
| eod_dayaggs_rest | not scheduled | — | Manual only. ~42k sequential REST calls for what day_aggs_v1 delivers in one file the next morning. |
On-demand: python -m ingest.entitlements, scripts/backfill.sh, scripts/repair.sh, python -m pricing.from_market. Archive rebuilds: scripts/build_term_structure.py, scripts/build_surface.py.
Not a job: decision_log is the append-only decision
record — schema in ingest.schemas, writer
ingest.common.decision_log.write_decisions. Nothing is
scheduled to produce it yet; the backtester writes first. A batch is
one session and one job, and the dataset can never be quarantined or
read as-of — decision history is never overwritten.
NY cron
Source of truth: deploy/schedule.json, one entry per unit
carrying the schedule twice — a cron form and a systemd
on_calendar form, machine-checked to fire at the same
instants (tests/test_schedule.py). Times are
America/New_York wall clock (the box's system timezone; timers elapse
in local time). The Ansible playbook templates a user-level
.service/.timer pair per entry into
~/.config/systemd/user/; every service still goes through
scripts/cronjob.sh, so a timer fire that overlaps a run
holding the job's flock logs job_skipped and exits 0.
Cutover: the crontab (deploy/crontab,
TZ=America/New_York) stays installed as a fallback while
the timers take over — safe because both paths share the lock, and the
timers' AccuracySec=1s keeps the loser's fire inside every
job's runtime (systemd's default 1-minute accuracy would not, and a
late duplicate snapshot_sweep would read as a stray
sweep). The crontab and its drift alarm are removed by a follow-up PR
after a clean day of timer runs. What is actually installed on the box,
and the drift alarms that keep it honest:
box operations.
Healthchecks: one check per job, slug massive-<job>
(underscores → hyphens). Ping key in .env; a leftover
HEALTHCHECKS_PING_URL without a ping key is rejected at
Settings.load (exit 2). Setup script
scripts/setup_healthchecks.py reads the same
deploy/schedule.json (CI asserts the job/check pairing).
Weekday session, Eastern time
Cash hours 09:30–16:00 ET. Websocket capture 09:25–16:35 ET — the 35-minute buffer is derived (options trade to 16:15 + 15-minute feed delay + 1-minute bar + 4-minute delivery margin), not picked. Bars are wall-clock occupancy, not CPU.
Entitlements
Probed, not assumed. Re-check with python -m ingest.entitlements
(exit 1 on drift in either direction: something stopped working, or
something newly available that no job captures). This absorbed the
former ingest.smoke, so it is also the post-bootstrap and
post-credential-change check.
Selected job-critical response bodies are shape-checked, so an endpoint covered by a validator that stops returning required fields counts as drift. The contract probe picks a fresh liquid contract each time rather than naming one, because a fixed ticker returns an entitled-but-empty 200 once it expires — indistinguishable from a revoked entitlement. Missing or placeholder S3 keys SKIP (exit 0) rather than FAIL, so a half-configured box reports "not set up yet" instead of a verdict about the plan.
- Entitled
- Contracts (incl. expired), full-chain snapshots with greeks/IV/OI (SPY, I:SPX, VIX), option trades, option aggs, T-1 equity minutes, grouped daily, dividends/splits, market status, delayed options WS, Treasury yields / inflation (
/fed/v1),trades_v1/minute_aggs_v1/day_aggs_v1. - Not entitled
- Option NBBO (REST and
quotes_v1), I:SPX / I:VIX (and other I:* index levels), equity trades/quotes,us_stocks_sip, same-day equity aggs,/v1/indicators/*.