Backtest overfitting
What the literature measured about selection bias, multiple testing and the gap between a backtest and what follows it.
What this shelf covers
A backtest picked from many candidates reports the best of a search, and the notes on this shelf measure how much of that best is luck. Three quantities recur: the number of trials behind a result, the length of the sample, and the correlation between candidates. The deflated Sharpe ratio uses the first two to raise the benchmark; the probability of backtest overfitting estimates how often the in-sample winner falls below the median out of sample; White's Reality Check and Hansen's SPA test the best rule against a benchmark after the search. Cohort studies compare backtests with the live results of the same strategies.
20 notes
Backtest versus Live on a Large Cohort of Algorithms
Does Publication Destroy Stock Return Predictability?
Evaluating Trading Strategies: What Threshold Should a Reviewer Demand
Evidence-Based Technical Analysis: Rules Tested Against Their Own Null
False Strategy Theorem (Expected Maximum Sharpe under N trials)
Hansen's Test for Superior Predictive Ability (SPA)
Lucky Factors: Testing a Candidate Against What Is Already Known
The Sharpe Ratio Can Be Manufactured by Selling Tails
Minimum Backtest Length (MinBTL) & Minimum Track Record Length (MinTRL)
Nothing Beats the Historical Mean Out of Sample, Except Under Constraints
Peeking at Accumulating Data Inflates the Error Rate
Ranking Noise and the Null of the Maximum
Statistical Overfitting: Why More Searching Gives a Worse Strategy
The Efficient-Market Null, Stated by Its Defenders
The Discount Between Backtest and Live, Measured at the Vendors
Where You Cut the Sample Is One More Free Parameter
Is There a Replication Crisis in Finance? Three Papers, Three Answers