[Library](https://stochastly.com/library)

# Backtest overfitting

What the literature measured about selection bias, multiple testing and the gap between a backtest and what follows it.

## What this shelf covers

A backtest picked from many candidates reports the best of a search, and the notes on this shelf measure how much of that best is luck. Three quantities recur: the number of trials behind a result, the length of the sample, and the correlation between candidates. The deflated Sharpe ratio uses the first two to raise the benchmark; the probability of backtest overfitting estimates how often the in-sample winner falls below the median out of sample; White's Reality Check and Hansen's SPA test the best rule against a benchmark after the search. Cohort studies compare backtests with the live results of the same strategies.

## 20 notes

[Backtest versus Live on a Large Cohort of Algorithms](https://stochastly.com/library/backtest-overfitting/backtest-versus-live-a-large-cohort)

[Deflated Sharpe Ratio (DSR)](https://stochastly.com/library/backtest-overfitting/deflated-sharpe-ratio)

[Deflation vs Selection Bias](https://stochastly.com/library/backtest-overfitting/deflation-vs-selection-bias)

[Does Publication Destroy Stock Return Predictability?](https://stochastly.com/library/backtest-overfitting/does-publication-destroy-predictability)

[Evaluating Trading Strategies: What Threshold Should a Reviewer Demand](https://stochastly.com/library/backtest-overfitting/evaluating-trading-strategies-the-threshold)

[Evidence-Based Technical Analysis: Rules Tested Against Their Own Null](https://stochastly.com/library/backtest-overfitting/evidence-based-technical-analysis-aronson)

[False Strategy Theorem (Expected Maximum Sharpe under N trials)](https://stochastly.com/library/backtest-overfitting/false-strategy-theorem)

[Hansen's Test for Superior Predictive Ability (SPA)](https://stochastly.com/library/backtest-overfitting/hansen-spa-test)

[Lucky Factors: Testing a Candidate Against What Is Already Known](https://stochastly.com/library/backtest-overfitting/lucky-factors-testing-against-what-exists)

[The Sharpe Ratio Can Be Manufactured by Selling Tails](https://stochastly.com/library/backtest-overfitting/manipulation-proof-performance-measures)

[Minimum Backtest Length (MinBTL) & Minimum Track Record Length (MinTRL)](https://stochastly.com/library/backtest-overfitting/minimum-backtest-length-and-track-record)

[Nothing Beats the Historical Mean Out of Sample, Except Under Constraints](https://stochastly.com/library/backtest-overfitting/nothing-beats-the-historical-mean-out-of-sample)

[Peeking at Accumulating Data Inflates the Error Rate](https://stochastly.com/library/backtest-overfitting/peeking-at-accumulating-data-inflates-error)

[Ranking Noise and the Null of the Maximum](https://stochastly.com/library/backtest-overfitting/ranking-noise-and-the-null-of-the-maximum)

[Statistical Overfitting: Why More Searching Gives a Worse Strategy](https://stochastly.com/library/backtest-overfitting/statistical-overfitting-and-backtest-performance)

[The Efficient-Market Null, Stated by Its Defenders](https://stochastly.com/library/backtest-overfitting/the-efficient-market-null-stated-by-its-defenders)

[The Discount Between Backtest and Live, Measured at the Vendors](https://stochastly.com/library/backtest-overfitting/the-live-discount-on-alternative-beta)

[Where You Cut the Sample Is One More Free Parameter](https://stochastly.com/library/backtest-overfitting/the-sample-division-is-a-free-parameter)

[Is There a Replication Crisis in Finance? Three Papers, Three Answers](https://stochastly.com/library/backtest-overfitting/three-answers-to-the-replication-question)

[Why Most Published Research Findings Are False](https://stochastly.com/library/backtest-overfitting/why-most-published-findings-are-false)

---
Source: https://stochastly.com/library/backtest-overfitting
