# Multiple testing in trading

What is the chance that at least one of 20 useless strategies passes a 5 percent significance test?

If the 20 tests are independent and no strategy has an edge, the probability of at least one false positive is 1 - 0.95^20, about 64 percent. Correlated strategies change that number, which is why resampling tests such as White's Reality Check exist.

Selecting a strategy after many backtests changes the meaning of a single unadjusted significance test. The tested family and its trial count matter.

## The family of trials

A trial includes a candidate whose result informed selection, even if it was discarded. Under twenty independent true null hypotheses tested at a 5% false-positive threshold, the chance of at least one false rejection is 1 - 0.95^20 = 0.6415, or about 64.2%. Independence is assumed for this illustration; correlated strategies require a dependence-aware analysis.

```
Family-wise false-positive probability = 1 - (1 - alpha)^N, under independent null tests
```

## One benchmark against many rules

White's Reality Check uses resampling to test whether the best candidate improves on a benchmark after data snooping. Hansen's superior predictive ability test changes the statistic and treatment of poor alternatives to improve power in the setting studied. The candidate family must be specified; omitting failed trials changes the question.

## Several candidate claims

Romano-Wolf StepM applies a resampling-based stepwise procedure to a family of hypotheses while controlling the family-wise error rate. It can identify several surviving hypotheses. White's and Hansen's tests instead address an overall best-versus-benchmark question.

## Where DSR fits

The deflated Sharpe ratio evaluates an observed Sharpe against a selection-adjusted benchmark based on the trial count and return moments. It is a separate summary for a selected strategy, not the p-value from Reality Check, SPA or StepM. Stochastly records a trial count for DSR and has multiplicity procedures in its validation code; inspect the run output to determine which procedure executed.

## How high the best of fifty looks by chance

Suppose 50 rule variants with no edge are each backtested on three years of independent daily returns. The annualized Sharpe ratio of each variant is then close to normal around zero, with a standard error near 1 / sqrt(3) = 0.58. The expected largest of 50 standard normal draws is about 2.25, so the best variant should show an annualized Sharpe near 1.3 by chance alone. A Sharpe of 1.3 reported after a 50-variant search is what the null hypothesis predicts. The deflated Sharpe ratio uses the same expected-maximum argument to set its benchmark.

```
E[max of N standard normals] ≈ (1 − γ) · Φ⁻¹(1 − 1/N) + γ · Φ⁻¹(1 − 1/(N·e)), γ ≈ 0.5772
```

## Frequently asked questions

### Does a conventional p-value cover a parameter search?

A single-test p-value does not include the selection over other tried parameters.

### What does StepM add to a global test?

It evaluates a family of individual hypotheses with stepwise family-wise error control.

### Can DSR replace Reality Check?

They test different summaries and require different inputs. Report the procedure that matches the registered question.

## Sources

White (2000). [A Reality Check for Data Snooping](https://doi.org/10.1111/1468-0262.00152). Econometrica 68(5), 1097-1126.

Hansen (2005). [A Test for Superior Predictive Ability](https://doi.org/10.1198/073500105000000063). Journal of Business & Economic Statistics 23(4), 365-380.

Romano and Wolf (2005). [Stepwise Multiple Testing as Formalized Data Snooping](https://doi.org/10.1111/j.1468-0262.2005.00615.x). Econometrica 73(4), 1237-1282.

Bailey and López de Prado (2014). [The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality](https://doi.org/10.3905/jpm.2014.40.5.094). Journal of Portfolio Management 40(5), 94-107.

[Primary source for Multiple testing in trading](https://doi.org/10.1111/1468-0262.00152)

## In the library

Data Snooping, P-Hacking & the Garden of Forking Paths

Multiple Testing: FWER vs FDR (Bonferroni/Holm, Benjamini–Hochberg, Storey)

Romano–Wolf StepM (Stepwise Multiple Testing)

White's Reality Check

[Hansen's Test for Superior Predictive Ability (SPA)](https://stochastly.com/library/backtest-overfitting/hansen-spa-test)

Harvey–Liu Haircut Sharpe & the Cross-Section of Expected Returns

## Related

[Deflated Sharpe ratio calculator](https://stochastly.com/tools/deflated-sharpe-ratio-calculator)

[Minimum track record length calculator](https://stochastly.com/tools/minimum-track-record-length-calculator)

[Deflated Sharpe ratio](https://stochastly.com/learn/deflated-sharpe-ratio)

White's Reality Check

---
Source: https://stochastly.com/learn/multiple-testing-in-trading