What happened? You fell into the multiple testing problem - the most pervasive, expensive, and under-discussed pitfall in quantitative finance. This chapter explains the problem rigorously and prepares the ground for the Bonferroni correction in Chapter 15. The 95% confidence threshold is defined to falsely reject the null hypothesis (declare a real effect when there is none) with probability 5% on any single test. If you run one test, the probability of a false positive is 5%. If you run 100 tests, even if every null hypothesis is true (no feature is genuinely predictive), the expected number of false positives is 5. If you run 1,000 tests, the expected number is 50.
Figure 1. Concept map for this episode: fight, desk translation, what to lock.
The arithmetic of testing many hypotheses
The 95% confidence threshold is defined to falsely reject the null hypothesis (declare a real effect when there is none) with probability 5% on any single test. If you run one test, the probability of a false positive is 5%. If you run 100 tests, even if every null hypothesis is true (no feature is genuinely predictive), the expected number of false positives is 5. If you run 1,000 tests, the expected number is 50. So in a search across 1,000 random features, you expect about 50 features to pass the 5% significance threshold purely by chance. None of them are actually predictive. They are noise. Deploy them and you will lose money.
The classic illustration
Suppose 10,000 stocks are listed somewhere in the world. Every Monday for ten years, a stock picker predicts "this stock will outperform" or "this stock will underperform." After ten years (520 weeks), by pure chance, some pickers will have been right 70% of the time. Are those pickers skilled? By the binomial distribution, the probability that a random coin-flipper achieves at least 364 wins out of 520 (70%) is vanishingly small - roughly 1.7 in 1020, or about one in fifty quintillion. So out of 10,000 random pickers, we expect zero to hit 70% by chance. Any picker who hits 70% is, plausibly, skilled.
Family-wise error rate vs false discovery rate
The statistics literature has developed two main frameworks for handling multiple testing. Family-Wise Error Rate (FWER) control aims to keep the probability that any of the discoveries are false below some chosen bound. The simplest FWER-controlling procedure is the Bonferroni correction (Chapter 15), which divides the per-test significance threshold by the number of tests: instead of testing each feature at the usual 5% threshold, you test each one at a much stricter threshold - the usual 5% split evenly across every test you run.
Backtest overfitting
The multiple testing problem in pure feature research is bad enough. But it gets dramatically worse in backtest-driven strategy research, where the researcher iteratively tunes strategy parameters by re-running backtests and selecting the best-performing configuration. Each tuning step is a hidden test, and the cumulative number of tests can be enormous. David Bailey, Marcos López de Prado, and co-authors have written extensively on this problem. Their key result: a researcher who runs many backtests on essentially-random strategies and selects the best-performing one can expect the in-sample Sharpe to be many times larger than the true out-of-sample Sharpe - roughly 1.7x inflated after 10 backtests, 2.5x after 100, and 3.3x after 1,000.
Figure. Deflated-Sharpe inflation grows with search size K — 1.7x at K=10, 2.5x at K=100, 3.3x at K=1,000.
Definition
The probability that at least one rejected null hypothesis in a family of tests is a false rejection (a false positive). Controlling FWER at some chosen level means the probability of any false positive across all the tests together is kept at or below that same level.
Worked picture
Test 1,000 features that are pure noise (none of them really predicts anything), each at the usual 5% significance threshold. Expected false positives: 50. Probability of at least one false positive: essentially certain - well above 99.9%. If you publish only the ten features with the strongest-looking results, all ten can be ghosts. The detective who searches a thousand alleys and reports only the one with footprints must ask: how many alleys had random mud?
What breaks in production
You design a strategy. The backtest looks mediocre. You tune the entry threshold; the Sharpe improves. You tune the stop level; the Sharpe improves further. You add a momentum filter; better. You restrict to only high-volume names; even better. After 30 minutes of tuning, the backtest looks excellent. You have just run roughly 30 implicit hypothesis tests, each conditioned on the previous one. Your effective K is somewhere between 30 and 1000 (depending on the search space). The inflation factor on the Sharpe is 2x to 3x. The "excellent" backtest is probably mediocre out of sample. The fix: separate research from validation. Do all the tuning on a training slice of the data. Hold out a test slice that is never touched during research. Report the Sharpe on the held-out test slice as the honest estimate. If the test-slice Sharpe is much worse than the training Sharpe, you have overfit.
The multiple testing problem
The probability that at least one of several independent tests produces a false positive climbs fast as you run more tests. With a single test at the usual 5% threshold, the false-positive chance is just 5%. Run 14 tests, and it's already 50% - you are more likely than not to find at least one spurious "significant" feature. Run 100 tests, and it's 99.4%. Running many tests at a fixed significance threshold guarantees false positives. The more tests you run, the more guaranteed.