Regime change in backtesting
Would the strategy still work if its market regime changed after selection?
A CUSUM test on regression residuals can flag instability under its assumptions, and a later holdout tests whether the chosen model still works. Neither result identifies the next market regime before it occurs.
CUSUM detects instability under an OLS model; it does not predict a future regime.
A regime change alters the distribution that generated a backtest segment. A pooled result can conceal a loss concentrated in a later market state.
Distribution and parameter shifts
Mean returns, volatility, correlations and the fitted response of a strategy can differ across periods. Ang and Timmermann survey persistent and abrupt changes in financial variables. A Markov-switching model treats the state as latent; a dated event split uses an observed boundary. Neither method establishes a future state in advance.
A segmented return example
Consider eight equally weighted monthly strategy returns: 2%, 2%, 2%, 2%, -1%, -1%, -1%, -1%. The pooled mean is 0.5% per month. The first four-month mean is 2%, and the last four-month mean is -1%, a decline of 3 percentage points. A pooled positive mean therefore does not describe performance in the later segment. The example is arithmetic, not evidence of a market regime.
Pooled mean = (4 × 2% + 4 × -1%) / 8 = 0.5%Tests and validation
Plot rolling estimates and predeclare candidate split dates or a break-detection rule. A CUSUM test of regression residuals can test parameter stability under its assumptions. Fit model parameters on training periods and evaluate later periods without using their outcomes to set the split. Repeatedly selecting a split from the same history changes the test's error rate.
Reading Stochastly results
The implemented regime readers describe aggregate run profit and loss; they do not filter entries or resize positions. The causal HMM state filter is marked roadmap. Compare out-of-sample results by dated segment, record parameter settings and inspect whether adverse performance is concentrated after a break. A regime label fitted retrospectively is not a live forecast.
Frequently asked questions
Does a positive pooled backtest cover each regime?
No. Report performance by dated segment and inspect whether the fitted parameters remain stable.
Can a break test name the next regime?
A historical break test evaluates changes in the observed series under a specified model. It does not identify a future state.
Why predeclare split dates?
Choosing the split after seeing strategy returns adds a selection step that changes the interpretation of the reported test.
Sources
Ang and Timmermann (2012). Regime Changes and Financial Markets. Annual Review of Financial Economics 4, 313-337.
Hamilton (1989). A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle. Econometrica 57(2), 357-384.
statsmodels (n.d.). breaks_cusumolsresid. Official statistical diagnostics documentation.
In the library
Regime Changes in Financial Markets (Ang-Timmermann Review)
Regime-Switching Models (Markov Switching, HMM)