Parameter robustness in backtesting
by Mahé Chadeffaud, founder of Stochastly
How do you tell a robust parameter value from a lucky peak on a backtest grid?
Average the score at the chosen value with its neighbors on the grid. In the page's example, strategy A keeps 94 percent of its peak Sharpe over the neighborhood and strategy B keeps 45 percent, so B's peak depends on one exact lookback.
A smooth neighborhood still needs out-of-sample evidence and a correction for the number of configurations searched.
A parameter value is robust when nearby values produce similar results. A score that collapses one grid step away from the selected value is more likely to reflect a feature of the sample.
Plateau and isolated peak
Plot a performance measure against one or two strategy parameters. A plateau is a region where neighboring values give similar scores. An isolated peak is a single high value surrounded by much lower ones. On noisy returns the highest cell of a search grid partly reflects noise, and its neighborhood shows how much of the score depends on that exact value.
Grid spacing changes the picture. A step that is large relative to the scale of the parameter can hide a narrow peak, and a very small step makes adjacent cells nearly identical by construction.
A neighborhood calculation
Take a moving-average lookback tested at 20, 30, 40, 50 and 60 bars, with a backtest Sharpe ratio for each value. Strategy A scores 0.9, 1.0, 1.1, 1.0 and 0.9. Strategy B scores 0.1, 0.2, 1.4, 0.3 and 0.1, so B has the higher peak.
Averaging each peak with its two neighbors gives (1.0 + 1.1 + 1.0) / 3 = 1.03 for A and (0.2 + 1.4 + 0.3) / 3 = 0.63 for B. The neighborhood average retains 94% of the peak of A and 45% of the peak of B.
Neighborhood average at p = mean of the scores at p and at its adjacent grid valuesSensitivity and the search
A smooth surface does not correct for the number of configurations tried. Bailey, Borwein, López de Prado and Zhu argue that overfitting a strategy to perform well in sample is relatively simple on a given financial series. Their probability of backtest overfitting (PBO), estimated with combinatorially symmetric cross-validation (CSCV), measures how often the in-sample winner of a candidate set ranks below the median out of sample. White's Reality Check tests the best rule of a search against a benchmark after data snooping.
Steegen, Tuerlinckx, Gelman and Vanpaemel propose a multiverse analysis, which repeats an analysis across reasonable alternative processing choices and reports how the conclusion varies.
“A multiverse analysis offers an idea of how much the conclusions change because of arbitrary choices in data construction and gives pointers as to which choices are most consequential in the fragility of the result.”
In Stochastly
The result of a parameter search includes a heatmap of the median out-of-sample Sortino ratio over the two numeric parameters with the most distinct values, drawn from configurations that cleared the run's gates and capped at 16 values per axis. It requires at least 6 such configurations with an out-of-sample Sortino and two varied parameters. A walk-forward run draws a grid per window from selection scores computed on the bars before that window.
The PBO path reports its CSCV splits. The multiverse analysis node reruns the configured pipeline over a declared grid of validation choices, including in-sample and out-of-sample shares, purge, embargo and walk-forward lengths, and reports robust and fragile fractions. It is a sensitivity analysis of the validation design and gives no independent evidence of significance.
Frequently asked questions
What is parameter robustness?
Stability of a backtest result when a parameter moves to nearby values. It is measured on a grid around the selected value.
Does a plateau prove an edge?
No. A plateau can come from a feature shared across the sample period, and a smooth surface still needs out-of-sample evidence and a correction for the search.
How does parameter sensitivity relate to PBO?
Sensitivity examines the neighbors of one selected value. PBO uses CSCV splits to estimate how often the in-sample winner of the candidate set falls below the median out of sample.
Sources
Bailey, Borwein, López de Prado and Zhu (2014). Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society 61(5), 458-471.
Bailey, Borwein, López de Prado and Zhu (2017). The Probability of Backtest Overfitting. Journal of Computational Finance 20(4), 39-69.
Steegen, Tuerlinckx, Gelman and Vanpaemel (2016). Increasing Transparency Through a Multiverse Analysis. Perspectives on Psychological Science 11(5), 702-712.
White (2000). A Reality Check for Data Snooping. Econometrica 68(5), 1097-1126.