Headless box — operations outside the repo

Companion to the system handbook (docs/index.html). The repo declares what the system should be; this page records the machine state it cannot declare — secrets, the installed schedule, services, and the external accounts — and how each piece was put in place. Host: brad-headless-server, user brad-lasater, no passwordless sudo (by design; everything below runs as the user).

Layout the repo assumes

Secrets (.env)

The Healthchecks management API key (hcw_…) is deliberately not stored anywhere on the box or in the repo; it is needed only when re-running scripts/setup_healthchecks.py --api-key ….

Installed schedule

active The canonical schedule is deploy/schedule.json — one entry per job, each carrying the schedule twice (cron and on_calendar forms, machine-checked against each other in CI). The Ansible playbook (deploy/ansible/playbook.yml) templates a user-level .service/.timer pair per entry into ~/.config/systemd/user/ and enables the timers.

The box crontab is still installed directly from the repo (crontab deploy/crontab — the placeholder path already matches the box user, no sed rewrite needed) as a fallback while the timers take over. The overlap is safe: both paths go through scripts/cronjob.sh and share /tmp/massive-<job>.lock, so whichever fires first runs and the other logs job_skipped and exits 0 — and every timer sets AccuracySec=1s, so the loser fires inside the winner's runtime rather than a minute later. After a clean day of timer runs, a follow-up PR removes the crontab and this paragraph. The box.yml “Crontab drift” step fails the build if the installed crontab diverges from the repo; that alarm fired on 2026-09-01 when PRs #10/#11 added rates_sync and ibkr_executions before the crontab was re-installed. Fix is always the same: pull main, re-install the crontab.

Systemd job timers

active One massive-<job>.timer + massive-<job>.service pair per schedule entry, generated — never hand-edited on the box; change deploy/schedule.json and re-run the playbook. Variants get their own unit but share the job's lock and Healthchecks check: massive-contracts-sync-expired runs contracts_sync --expired, massive-snapshot-sweep-eod runs snapshot_sweep --eod.

• Type=oneshot services through scripts/cronjob.sh; logs still append to /data/massive/logs/cron.log, and journald keeps unit start/stop/exit status — journalctl --user -u massive-coverage-audit.
• AccuracySec=1s + Persistent=false + RandomizedDelaySec=0 on every timer. The 1s accuracy is load-bearing for the crontab overlap (systemd's default is 1 min — a timer firing up to 60s late would double-capture snapshot_sweep); no persistence because a missed sweep is worthless an hour later and catch-up would stampede the day's backlog after a reboot.
• Restart=on-failure with StartLimitBurst=3 / StartLimitIntervalSec=1800 only on low-frequency jobs — cross-run retry, the thing cron never had. The 1/min and 5/min jobs get no retry: the next tick is closer than any sane delay.
• OnFailure=massive-alert@<check-slug>.service pings the check's /fail for deaths the job cannot report itself (OOM kill, unit start failure, exhausted restarts). Ordinary non-zero exits already ping /fail from the job.
• Timers elapse in the box's local time, which must stay America/New_York — the playbook fails loudly if the system timezone drifts.

systemctl --user list-timers 'massive-*'     # next/last fire per job
journalctl --user -u massive-snapshot-sweep -n 50
ansible-playbook -i deploy/ansible/inventory_local.ini deploy/ansible/playbook.yml  # (re)converge, on the box
ansible-playbook -i deploy/ansible/inventory.ini deploy/ansible/playbook.yml        # same, over ssh from a workstation
ansible-playbook deploy/ansible/render_units.yml  # render to /tmp only, no install

GitHub Actions self-hosted runner

active Installed 2026-09-01 to replace Buildkite. Registered to the repo as ingest-box (labels self-hosted, linux, x64 — what .github/workflows/box.yml targets). Runs as a user-level systemd unit, no sudo:

~/.config/systemd/user/actions-runner.service
  ExecStart = ~/actions-runner/run.sh        # foreground mode; runsvc.sh needs sudo
  EnvironmentFile = ~/crack-the-sky/.env # so jobs see the credentials
  Restart = always
loginctl enable-linger brad-lasater          # survives logout/reboot

Verify from anywhere: gh api repos/{owner}/{repo}/actions/runners should show ingest-box online. Accepted risk (documented in the workflow header): pushed code runs next to the market-data credentials and /data/massive — fine for a private, single-owner repo with 2FA; the keys are read-only and no brokerage access is reachable.

Healthchecks.io

active Twenty-two user-timer units, twenty unique jobs. The expired-contracts and EOD-snapshot passes share their parent job's lock and Healthchecks check. That leaves twenty scheduled checks named massive-<job>, plus two extra checks pinged by a running job rather than by cron — massive-ws-minute-bars-alive and massive-snapshot-sweep-all-chains — twenty-two checks in all. Jobs ping via the ping key in .env; schedules and grace periods come from deploy/schedule.json via scripts/setup_healthchecks.py (without that step, auto-created checks get a default daily schedule and false-alarm over weekends on the weekday-only jobs, and every day for monthly prune). A notification channel (email/Slack) must exist in the Healthchecks UI or alerts go nowhere.

IBKR Flex Web Service

active Verified live 2026-09-01: token + query id authenticate, the statement generates, the period guard refuses a mislabeled date, and a no-fills day exits cleanly. The query’s saved period (“Last Business Day”) decides the statement’s contents — the 18:30 ET cron passes no --date and the job checks the statement period matches. If the token is regenerated (~yearly), only .env changes.

Removed: Buildkite

gone Repo webhook deleted 2026-09-01; no buildkite-agent unit or process exists on the box. The only remaining artifact is the subscription itself — cancelled (or cancelled separately) in the Buildkite UI, which is where billing stops. Replacement: the self-hosted runner above plus a GitHub ruleset on main (PR required; test (3.11) / test (3.12) checks required).

Bring-up calendar

• 2026-08-31 — first capture day; partial (sweeps began 17:25 ET), so coverage_audit and the box workflow correctly read red for it.
• 2026-09-01 — first full session at minute cadence (SPY + I:SPX + VIX); runner installed; crontab aligned; IBKR Flex wired; Healthchecks schedules set; first real drift_check canary at 17:00 ET.

Verify-it-works cheat sheet

# runner online?
gh api repos/{owner}/{repo}/actions/runners --jq '.runners[] | {name,status}'

# schedule matches the repo?
crontab -l | diff - deploy/crontab
systemctl --user list-timers 'massive-*'

# entitlements + payload shapes, live?
venv/bin/python -m ingest.entitlements

# box services
systemctl --user status actions-runner