# Probability of backtest overfitting (PBO)

What does a PBO score of 0.6 mean for the way a strategy configuration was chosen?

In 60 percent of the CSCV splits, the configuration ranked first on the in-sample blocks finished below the median of all candidates on the held-out blocks. Picking the in-sample winner did worse than picking a configuration at random.

PBO estimates the probability that the configuration which looked best in sample ranks below the median out of sample.

## The idea

When a researcher tests many configurations and keeps the best one, the selection itself can be overfit. PBO, introduced by Bailey, Borwein, López de Prado and Zhu, measures how often that selection fails on data it did not use.

[Screenshot: CPCV path distribution in Stochastly: out-of-sample results across recombinations of purged blocks of the same backtest.](https://stochastly.com/screenshots-current/learn-probability-of-backtest-overfitting--cpcv-path-distribution--desk.png)

CPCV path distribution in Stochastly: out-of-sample results across recombinations of purged blocks of the same backtest.

## How it is computed: CSCV

Combinatorially symmetric cross-validation splits the matrix of returns of all configurations into S blocks of time. For each way of choosing half of the blocks as in-sample, the best configuration in sample is located, and its rank among all configurations on the remaining blocks is recorded.

Each rank is turned into a logit. PBO is the share of combinations where the logit is below zero, meaning the in-sample winner landed in the bottom half out of sample.

When several configurations tie for the best in-sample score, Stochastly counts the combination as the average over the tied winners, the expected value of picking one of them at random. Identical configurations are counted once.

```
λ = ln(ω / (1 − ω)), PBO = share of combinations with λ < 0
```

“We illustrate this framework with specific generic, model-free and nonparametric implementations in the context of investment simulations; we call these implementations combinatorially symmetric cross-validation (CSCV).”

Bailey, Borwein, López de Prado and Zhu (2017), [The Probability of Backtest Overfitting](https://doi.org/10.21314/JCF.2016.322)

## Reading the score

A PBO close to 0 means the selection process tends to pick configurations that hold up. A PBO around 0.5 means picking the in-sample winner does no better than picking at random. Above 0.5, the selected configurations tend to do worse out of sample.

PBO judges the search process, so it needs several configurations. With a small number of configurations its estimate is noisy.

## Frequently asked questions

### What does a PBO of 0.5 mean?

The best configuration in sample has even odds of falling below the median out of sample: the selection adds no information.

### What is a good PBO?

Lower is better. Values well below 0.5 across several splits indicate that the choice of configuration generalizes.

### Is PBO the same as the deflated Sharpe ratio?

No. PBO evaluates the ranking of many configurations; the deflated Sharpe ratio corrects one Sharpe ratio for the number of trials. They answer related questions and are often reported together.

## Sources

Bailey, Borwein, López de Prado and Zhu (2014). [Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance](https://doi.org/10.1090/noti1105). Notices of the American Mathematical Society 61(5), 458-471.

Bailey, Borwein, López de Prado and Zhu (2017). [The Probability of Backtest Overfitting](https://doi.org/10.21314/JCF.2016.322). Journal of Computational Finance 20(4), 39-69.

Bailey and López de Prado (2014). [The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality](https://doi.org/10.3905/jpm.2014.40.5.094). Journal of Portfolio Management 40(5), 94-107.

Bailey and López de Prado (2012). [The Sharpe Ratio Efficient Frontier](https://doi.org/10.21314/JOR.2012.255). Journal of Risk 15(2), 3-44.

Harvey, Liu and Zhu (2016). [… and the Cross-Section of Expected Returns](https://doi.org/10.1093/rfs/hhv059). Review of Financial Studies 29(1), 5-68.

[Primary source for Probability of backtest overfitting (PBO)](https://doi.org/10.21314/JCF.2016.322)

## In the library

Probability of Backtest Overfitting (PBO / CSCV)

## Related

[PBO calculator](https://stochastly.com/tools/probability-of-backtest-overfitting-calculator)

[Deflated Sharpe ratio](https://stochastly.com/learn/deflated-sharpe-ratio)

[Overfitting in trading](https://stochastly.com/learn/overfitting-in-trading)

[Stochastly validation method](https://stochastly.com/research/method)

[Deflated Sharpe ratio calculator](https://stochastly.com/tools/deflated-sharpe-ratio-calculator)

---
Source: https://stochastly.com/learn/probability-of-backtest-overfitting