# Overfitting in trading

How can a trader tell whether a backtested strategy is overfit to its sample?

Compare in-sample with out-of-sample results, move each parameter one grid step and check that performance degrades gradually, and correct the Sharpe ratio for the number of variants tried. A deflated Sharpe ratio near zero or a PBO above 0.5 points to overfitting.

An overfit strategy has learned the noise of its backtest. It fits the past closely and loses its edge on data it has not seen.

## What overfitting means for a strategy

Overfitting, also called curve fitting, happens when the rules of a strategy are tuned so closely to one historical sample that they describe its accidents. The backtest looks excellent because the parameters were chosen on that same data.

Each free parameter, filter or exit rule adds a degree of freedom. With enough of them, a backtest can reproduce almost any equity curve, including one that has no predictive value.

“given any financial series, it is relatively simple to overfit an investment strategy so that it performs well IS.”

Bailey, Borwein, López de Prado and Zhu (2014), [Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance](https://doi.org/10.1090/noti1105)

[Screenshot: The Luck cloud chart in Stochastly: a strategy's equity curve against strategies that guessed direction at random with the same sizes.](https://stochastly.com/screenshots-current/learn-overfitting-in-trading--luck-cloud-vs-random-strategies--desk.png)

The Luck cloud chart in Stochastly: a strategy's equity curve against strategies that guessed direction at random with the same sizes.

## Why trying many variants inflates results

Testing many variants and keeping the best is a multiple-testing problem. Among 100 random strategies with no edge, the best one will still show an attractive Sharpe ratio by chance.

The expected maximum Sharpe ratio grows with the number of trials. Bailey and López de Prado show that the more configurations a researcher tries, the longer the backtest must be before its best result means anything.

```
E[max SR] ≈ sqrt(V[SR]) · ((1 − γ) · Φ⁻¹(1 − 1/N) + γ · Φ⁻¹(1 − 1/(N·e)))
```

“Given this extensive data mining, it does not make sense to use the usual criteria for establishing significance.”

Harvey, Liu and Zhu (2016), [… and the Cross-Section of Expected Returns](https://doi.org/10.1093/rfs/hhv059)

## Signs that a backtest is overfit

A strategy with many parameters and few trades, a performance that collapses when one parameter moves slightly, a large gap between in-sample and out-of-sample results, or an equity curve that is smoother than the market it trades.

A robust strategy degrades gradually when its parameters change and keeps a positive expectancy on periods that were not used to design it.

## How to measure it

Out-of-sample testing and walk-forward analysis check the strategy on unseen data. When selecting a winner from a population of configurations, the deflated Sharpe ratio adjusts for that selection. The probability of backtest overfitting (PBO) estimates how often the in-sample winner falls below the median out of sample.

When the required data are available, Stochastly reports these diagnostics and explains the run status, including NO TRADES or NOT CONCLUSIVE. Previews are labelled separately.

## Frequently asked questions

### How do I know if my trading strategy is overfit?

Compare in-sample and out-of-sample performance, perturb each parameter and check that results degrade smoothly, and correct the Sharpe ratio for the number of variants you tried. A deflated Sharpe ratio close to zero or a PBO above 0.5 indicates overfitting.

### Is curve fitting the same as overfitting?

In trading the two terms are used for the same problem: rules tuned to one historical sample so closely that they capture its noise.

### How many parameters are too many?

It depends on the number of independent trades. A common rule of thumb is to keep several dozen trades per free parameter, and to count every rule you tested and discarded as a trial.

## Sources

Bailey, Borwein, López de Prado and Zhu (2014). [Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance](https://doi.org/10.1090/noti1105). Notices of the American Mathematical Society 61(5), 458-471.

Harvey, Liu and Zhu (2016). [… and the Cross-Section of Expected Returns](https://doi.org/10.1093/rfs/hhv059). Review of Financial Studies 29(1), 5-68.

Bailey, Borwein, López de Prado and Zhu (2017). [The Probability of Backtest Overfitting](https://doi.org/10.21314/JCF.2016.322). Journal of Computational Finance 20(4), 39-69.

Bailey and López de Prado (2014). [The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality](https://doi.org/10.3905/jpm.2014.40.5.094). Journal of Portfolio Management 40(5), 94-107.

White (2000). [A Reality Check for Data Snooping](https://doi.org/10.1111/1468-0262.00152). Econometrica 68(5), 1097-1126.

[Primary source for Overfitting in trading](https://doi.org/10.1090/noti1105)

## In the library

[Statistical Overfitting: Why More Searching Gives a Worse Strategy](https://stochastly.com/library/backtest-overfitting/statistical-overfitting-and-backtest-performance)

Financial data as low signal-to-noise (why ML overfits here)

[Backtest versus Live on a Large Cohort of Algorithms](https://stochastly.com/library/backtest-overfitting/backtest-versus-live-a-large-cohort)

Marcos Lopez de Prado (the toolkit this app is largely built on)

## Related

[Probability of backtest overfitting](https://stochastly.com/learn/probability-of-backtest-overfitting)

[Deflated Sharpe ratio](https://stochastly.com/learn/deflated-sharpe-ratio)

[Walk-forward analysis](https://stochastly.com/learn/walk-forward-analysis)

[Parameter robustness in backtesting](https://stochastly.com/learn/parameter-robustness-in-backtesting)

[How Stochastly checks a backtest](https://stochastly.com/product/guards)

---
Source: https://stochastly.com/learn/overfitting-in-trading