Point-in-Time Data, Revisions and Vintages
How should a backtest retain the historical version of revised economic data?
Store the observation value and the date when each version became public, then join signals to the latest version known at the decision timestamp. Later revisions can improve or worsen a historical result; their direction depends on the rule and sample.
Economic-data revisions can amplify or reduce a backtest, depending on the signal and sample.
Evidence map
| Aspect | Finding |
|---|---|
| What it is | A vintage records the values of a data series as published on a particular date. |
| Key result / practice | Orphanides shows that evaluating historical policy rules with revised rather than real-time macroeconomic data can change the inferred decisions. |
| Why it matters for backtesting | Purging and walk-forward splits protect against some forms of leakage between train and test periods. |
What it is
A point-in-time dataset keeps enough publication history to reconstruct which values were available at a past decision time. Some economic series are revised after first release. A backtest that reads a later revision at an earlier decision time uses information the trader did not yet have.
Key result / practice
Croushore and Stark assemble a real-time dataset to examine forecasts and policy questions using the values that were actually available then. Those studies establish that the vintage can matter; they do not establish that every revision has the same sign or that every series is revised. A release lag only fixes timing. If the researcher shifts the current final value to an earlier release date, the simulation can still import a later correction. Check the provider's documentation for each field: first release, scheduled or unscheduled revisions, historical archive start, and the difference between observation date and publication date. Both measured series and constructed indices may change later, so the label alone cannot determine revision risk.
Why it matters for backtesting
They do not repair values that were revised after a simulated decision. The direction of bias is not fixed: a revision can amplify or reduce a particular strategy's apparent performance depending on the signal, threshold and evaluation period. Compare decisions produced by historical vintages with those produced by today's final values on the same dates, and record every changed trade. If the archive starts later than the apparent full history, report the shorter usable point-in-time window. Keep model burn-in and missing releases visible. Do not silently substitute final data before the first archived vintage.
Worked decision
Suppose a quarterly release later changes from 1.8 to 2.2, and a strategy enters only above 2.0. Using 2.2 on the original release day creates a trade that the live rule could not have made. Another correction might move a value below the threshold and remove a profitable historical trade. Neither direction can be assumed in advance. The decision log should store the vintage identifier, release timestamp, raw value, threshold and resulting order for each trade.
Source
Croushore & Stark, "A Real-Time Data Set for Macroeconomists", Journal of Econometrics 105(1), 2001, 111-130; Croushore, "Frontiers of Real-Time Data Analysis", Journal of Economic Literature 49(1), 2011; Orphanides, "Monetary Policy Rules Based on Real-Time Data", American Economic Review 91(4), 2001. Primary source