The statistical firewall
Search on train, evaluate once on holdout, never peek at the sealed tail. · 10 min
Order of operations
Never optimise on the full history and call it validated.
- Chronological split by time (default 70/30).
- Search, tune orbs, pick variants — train only.
- BH / Bonferroni with the campaign-level
candidatesSearched. - Evaluate the chosen config once on holdout.
- Walk-forward folds report sign consistency, not a second search.
Method
Interfaces exist for Deflated Sharpe, PBO, Reality Check and SPA. They are not yet wired. Absence is documented rather than faked.
Key points
- LLM assistants may propose hypotheses and write summaries. They never compute statistics.
- All random procedures are seeded.
- Language: observed effect, historical association, out-of-sample result. Never “planet X will make Y rise”.
References
- White, A reality check for data snooping — White H., Econometrica 68(5), 2000, 1097–1126
- RAINBOW methodology — docs/RESEARCH-METHODOLOGY.md