# In-sample vs out-of-sample testing

How much of the price history should stay out of sample, and when can it be reused?

Common practice keeps the most recent 20 to 30 percent of the history aside, with enough trades in it to estimate performance. Each look at it followed by a change to the strategy turns that period into design data and counts as a trial.

The 20 to 30 percent share is a common convention with no theoretical optimum; the number of trades in the holdout matters more.

In-sample data is used to design and tune a strategy. Out-of-sample data is kept aside to test it once the design is fixed.

## Why the split matters

A strategy tends to look better on the data used to build it. The out-of-sample result is the estimate of how it behaves on new data, as long as that data had no influence on the design.

“A backtest is realistic when the IS performance is consistent with the OOS performance.”

Bailey, Borwein, López de Prado and Zhu (2014), [Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance](https://doi.org/10.1090/noti1105)

[Screenshot: Edge decay in Stochastly: the rolling mean result per bar compared with the level measured in the first half of the sample.](https://stochastly.com/screenshots-current/learn-in-sample-vs-out-of-sample--edge-decay-recent-vs-is--desk.png)

Edge decay in Stochastly: the rolling mean result per bar compared with the level measured in the first half of the sample.

## How to split

Keep the most recent period as out-of-sample, since markets change over time. Walk-forward analysis repeats the split on successive windows, and purged cross-validation removes observations near the boundary so that overlapping labels do not leak across it.

## Common leaks

Looking at out-of-sample results and then adjusting the strategy turns the test period into design data. Normalizing features on the full series, or choosing the universe with knowledge of the test period, also leaks information.

“Researchers have lived through the hold-out sample and thus understand the history, are knowledgeable about when markets rose and fell, and associate leading variables with past experience.”

Arnott, Harvey and Markowitz (2019), [A Backtesting Protocol in the Era of Machine Learning](https://doi.org/10.3905/jfds.2019.1.064)

## A worked split

Take ten years of daily data. Design and tune on the first seven and a half years, then run the frozen strategy once on the last two and a half. For independent daily returns, the standard error of an annualized Sharpe ratio near zero over 2.5 years is about 1 / sqrt(2.5) = 0.63. An out-of-sample Sharpe of 0.5 is therefore still compatible with no edge at the 95 % level, and a Sharpe of 1.5 is not. Write that threshold down before the single run.

If the strategy trades twice a month, the holdout contains about 60 trades. That is enough to compare the average trade with the in-sample period, and too few to judge rare events such as a gap through a stop.

## Frequently asked questions

### How much data should be out-of-sample?

Common splits keep 20 to 30 % of the history out of sample, with enough trades in that period to estimate performance.

### Can I reuse out-of-sample data?

Each reuse after seeing the results makes it part of the design. Count those looks as trials, or keep a final holdout that is used once.

## Sources

Bailey, Borwein, López de Prado and Zhu (2014). [Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance](https://doi.org/10.1090/noti1105). Notices of the American Mathematical Society 61(5), 458-471.

Arnott, Harvey and Markowitz (2019). [A Backtesting Protocol in the Era of Machine Learning](https://doi.org/10.3905/jfds.2019.1.064). Journal of Financial Data Science 1(1), 64-74.

Bailey, Borwein, López de Prado and Zhu (2017). [The Probability of Backtest Overfitting](https://doi.org/10.21314/JCF.2016.322). Journal of Computational Finance 20(4), 39-69.

Brown, Goetzmann, Ibbotson and Ross (1992). [Survivorship Bias in Performance Studies](https://doi.org/10.1093/rfs/5.4.553). Review of Financial Studies 5(4), 553-580.

[Primary source for In-sample vs out-of-sample testing](https://doi.org/10.1090/noti1105)

## In the library

[Nothing Beats the Historical Mean Out of Sample, Except Under Constraints](https://stochastly.com/library/backtest-overfitting/nothing-beats-the-historical-mean-out-of-sample)

Feature importance: MDA vs MDI + SHAP

"Real" and "tradable" are two separate verdicts, and cost is what separates them

[In Defence of Optimisation: the Fallacy of Equal Weighting](https://stochastly.com/library/risk-and-portfolio/in-defense-of-optimization-against-one-over-n)

## Related

[Walk-forward analysis](https://stochastly.com/learn/walk-forward-analysis)

[Purged k-fold and CPCV for finance](https://stochastly.com/learn/purged-k-fold-and-cpcv)

[Look-ahead bias](https://stochastly.com/learn/look-ahead-bias)

[Overfitting in trading](https://stochastly.com/learn/overfitting-in-trading)

---
Source: https://stochastly.com/learn/in-sample-vs-out-of-sample