How I intend to avoid fooling myself

Why pre-commit to an evaluation protocol before running a backtest?

The failure mode in systematic trading research is not bad mathematics. It is a thousand small, individually reasonable decisions — a slightly different lookback, a filtered universe, an excluded month — each made after seeing what it does to the equity curve. The result is a strategy fitted to its own evaluation, which performs beautifully until the moment real money is behind it.

The defence is to fix the judging criteria first and count every decision made afterwards. Everything below is that commitment, stated in public so that departing from it is costly.

What data and universe does the system trade, and how is survivorship bias avoided?

Phase 1 trades defined-risk SPY and SPX/SPXW structures at roughly 5–45 days to expiry, with VIX options captured for the term structure, and every input is reconstructed as it stood at the decision timestamp rather than as it stands today. Three rules make that concrete:

• Universe: defined-risk SPY and SPX/SPXW structures at roughly 5–45 days to expiry, with VIX options captured for the term structure. SPX and SPXW are European-style and cash-settled; SPY is American-style and physically settled, so early exercise is priced with an American model rather than assumed away. Liquidity screens are applied on the date of the decision rather than retrospectively; when the universe extends beyond the index complex, selecting today’s liquid names and testing them over history is survivorship bias wearing a lab coat.
• Point-in-time discipline: every input must be as it was available at the decision timestamp — including dividends, splits, index membership, and the discount curve. Any field that cannot be reconstructed point-in-time is excluded from the model.
• Quote hygiene, on the entitled inputs: static no-arbitrage bounds enforced — a price outside the discounted bounds raises rather than returning a NaN — open-interest filtering, and the forward implied from put–call parity rather than taken from a vendor field.
• Quote hygiene that is blocked, and why: dropping crossed and locked quotes and filtering by spread width are defined in terms of bid and ask. Option NBBO is not entitled on the current data tier, so no bid, ask or mid exists anywhere in the warehouse and every price is a traded price — last trade, else day close. These checks are specified and unimplemented, not implemented and unmentioned.

How do you cross-validate a trading strategy without leaking the future?

Standard k-fold cross-validation is invalid on overlapping financial labels, because a position held for three weeks shares information with three weeks of neighbouring observations. This system uses purged and embargoed k-fold for model selection, walk-forward for the reported number, and a final held-out period used exactly once:

• Purged k-fold with embargo. Options labels span time — a position held for three weeks overlaps three weeks of subsequent observations. Standard k-fold leaks that overlap directly into the validation set. Training observations whose label windows overlap the validation window are purged, and an embargo period after each validation fold is dropped entirely to handle serial correlation.
• Walk-forward as the headline. Cross-validation is for model selection. The number that gets reported is walk-forward: train on the past, trade the next period, roll forward, never look back.
• A held-out final period, untouched until the system is otherwise frozen, and used exactly once. If it disagrees with the development results, the development results were wrong.

What baselines must a volatility forecast beat before it is allowed to size a position?

A volatility model is never evaluated against zero. It is evaluated against the cheapest forecast that already works, and it must beat that forecast out of sample before it is allowed anywhere near a position. Four baselines are mandatory here:

• Implied volatility itself as the forecast of realised volatility.
• HAR-RV on high-frequency realised variance.
• A GARCH-family conditional variance model.
• For the strategy as a whole: naive short-variance carry, unconditionally sized.

The rule is a promotion rule, not a presumption that the more complex model must work: a learned model is promoted only where it demonstrates incremental out-of-sample value over the appropriate baseline. A model that fails to beat HAR-RV has produced a result — that on this data, at this horizon, the cheap forecast is the better one — and that result gets written up rather than debugged away. Most of them will be that.

How are execution costs and frictions modelled in an options backtest?

Fills are never assumed at the mid, because in options the spread is frequently larger than the entire modelled edge. Every backtested trade in this system is to be charged for the following, at minimum:

• Half-spread crossing, scaled by quoted width at the decision time and by order size relative to displayed depth. Not yet calibratable: quoted width and displayed depth are NBBO fields, and option NBBO is not entitled on the current data tier. With no bid or ask the crossing cost can be bounded — roughly half the spread in volatility terms — but it cannot be measured or corrected for.
• Commissions and exchange fees per contract.
• Legging risk on multi-leg structures.
• Delta-hedging costs at the actual rebalancing frequency, including the hedge’s own spread.
• SPY early-assignment and borrow costs (SPX/SPXW remain European-style and cash-settled).
• Margin requirements, modelled well enough to know whether a drawdown becomes a forced liquidation.

Every reported result states its assumed fill convention. A strategy whose edge disappears under a realistic spread assumption is reported as such rather than quietly re-run at the mid.

Which performance statistics are reported with every result?

Every result published from this system reports seven statistics together: raw Sharpe, deflated Sharpe, the trial count behind them, the probability of backtest overfitting, maximum drawdown and time to recover, turnover and cost share of gross P&L, and return attribution by Greek. A raw Sharpe ratio on its own is not a result — without knowing how many configurations produced it, it is uninterpretable.

What pre-committed conditions retire a strategy?

Kill criteria have to be fixed in advance, because the temptation to keep adjusting a losing strategy is the expensive one and it is strongest exactly when judgement is worst. Any one of the following retires a strategy from this system to the research log rather than the portfolio:

• Deflated Sharpe not distinguishable from zero at the confidence level fixed for the experiment.
• Live out-of-sample performance below the lower bound of the backtest’s predicted range for the underperformance window fixed for the experiment.
• Realised costs exceeding modelled costs by more than the cost margin fixed for the experiment — the model was wrong about the world, whatever the P&L says.
• Return attribution showing the P&L came from an exposure the strategy was not intended to take, even if that P&L is positive.

That last one matters most. A strategy that makes money for reasons other than its thesis is not a working strategy; it is an unmeasured risk that has not yet been paid for.

What gets published, including the failures?

Publication is not selective, because a research record that only contains successes is indistinguishable from one that has been curated after the fact. Four things go up regardless of how they reflect on the system. None has been triggered yet; they take effect with the first result:

• Every result with its full statistics table, never a headline number alone.
• Every dead hypothesis, with what was expected and what actually happened.
• Every material change to this protocol, dated, with the reasoning.
• Trade-level logs behind any live track record, so the reported statistics can be recomputed independently. Not yet backed: the published data contract stores daily aggregates and cannot express a trade, so this commitment requires a schema extension that has not been made. It is listed here because it was promised, and deleting it quietly would be worse than flagging it.

This protocol is versioned in the repository. Changes are commits, not edits — last revised September 7, 2026.