p-values, effect size, multiple testing
One pretty p-value is cheap when you searched many bodies, orbs, assets and horizons. · 10 min
What the Lab reports
- p — randomised-event (permutation) p, Welch fallback versus unconditional.
- d — Cohen’s d versus unconditional.
- adjusted p — Bonferroni or Benjamini–Hochberg using
candidatesSearched.
Multiple testing
Astro × market grids are a textbook family-wise error trap. candidatesSearched is campaign-level (or miner-run), not the size of one batch. The snooping penalty is 1 / (1 + log10(n)).
Method
Search on train, BH-correct, evaluate once on holdout. Never rank on the full history and call the result validated.
Key points
- p < 0.05 on one pre-registered test is not the same as p < 0.05 after 400 miner candidates.
- Sign flips on holdout cap the Rainbow Score regardless of in-sample p.
- Seeded RNG (
makeRng) makes permutation draws reproducible from the stored config.
References
- Benjamini and Hochberg FDR — Benjamini Y. and Hochberg Y., Journal of the Royal Statistical Society B 57(1), 1995, 289–300
- Bonferroni inequality — Bonferroni C.E., Teoria statistica delle classi e calcolo delle probabilità, 1936