Library / Machine learning in finance
Conformal Prediction: Intervals With a Guarantee and No Distributional Assumption
When does a conformal interval retain its promised coverage after market data change?
The finite-sample marginal coverage statement requires exchangeable observations, a condition financial series often violate through dependence and regime shifts. Calibrate on an untouched period, then report realized coverage by later period and regime. Treat any adapted time-series guarantee according to that method's stated assumptions.
The coverage guarantee is marginal under exchangeability; it is not conditional coverage in every regime.
Evidence map
| Aspect | Finding |
|---|---|
| What it is | A method for turning any prediction model into one that outputs a set or interval with a guaranteed coverage rate, requiring no more than that the data be exchangeable. |
| Key result / formula | The construction uses a nonconformity measure, a score saying how unusual a candidate outcome looks given the model and the past data. |
| Why it matters for backtesting | The attraction for backtesting is that the guarantee does not require the return distribution to be anything in particular, which is welcome given how badly financial returns satisfy the usual assumptions. |
What it is
It is unusual in offering a finite-sample guarantee where most methods give an asymptotic one, and it wraps around an existing model and leaves it in place.
Key result / formula
For a new observation, the candidate values whose nonconformity score is not extreme relative to the scores observed on held-out data are collected into the prediction set. The theorem is that this set contains the true outcome with at least the stated probability, in finite samples, for any underlying model and any data distribution, provided the observations are exchangeable. The price is that the guarantee is marginal rather than conditional: it holds on average over the data and need not hold within each region of the feature space, so intervals can be uninformatively wide where the model is uncertain and the coverage can be uneven across regimes. The split variant makes the computation cheap by using a single held-out calibration set.
Why it matters for backtesting
The obstacle is the exchangeability condition, which financial time series violate: returns are dependent and their distribution shifts, so the guarantee does not hold as stated. Two positions are defensible. Use conformal intervals as an uncertainty diagnostic, comparing realised coverage against nominal coverage on held-out periods, where a large shortfall is itself a measurement of non-stationarity. Or use the variants developed for shifted and dependent data, stating that the guarantee is then approximate. What an agent should not do is present the guarantee as intact on a financial series, since the assumption that carries it is the one the data breaks.
Source
Shafer & Vovk, "A Tutorial on Conformal Prediction", Journal of Machine Learning Research 9, 2008, 371-421. Primary source