Library / Machine learning in finance
Two Cultures, and No Free Lunch
How do statistical and algorithmic models fail differently on market data?
Choose a model for the decision at hand, then test prediction on future data and inspect whether its explanation is adequate for the intended use. Breiman contrasts data models with algorithmic prediction; Wolpert's no-free-lunch result averages across possible problems and supplies no universal ranking for market datasets.
Breiman does not imply black-box methods always fail; no-free-lunch does not force a formal prior for every method.
Evidence map
| Aspect | Finding |
|---|---|
| What it is | Three statements about what modelling is for, which together set the terms an agent should use when a user asks whether to prefer a statistical model or a machine learning one. |
| Key result / formula | Breiman contrasts two cultures. One assumes a data-generating model, fits it, and interprets its parameters; its virtue is interpretability and its risk is that conclusions depend on an assumption rarely tested. |
| Why it matters for backtesting | In Stochastly the two cultures correspond to two different questions, and users conflate them. |
What it is
The answer depends on the goal, and no algorithm is best in general.
Key result / formula
The other treats the mechanism as unknown and optimises predictive accuracy on held-out data; its virtue is that accuracy is measurable and its cost is that its predictions may require additional explanation for a causal question. Breiman's argument is that the first culture's dominance led statisticians to irrelevant problems, and that predictive accuracy is the proper arbiter. Wolpert's result constrains both: averaged over all possible problems, no learning algorithm outperforms any other, so a useful method must be evaluated against the distribution of problems actually faced. This theorem does not require a formal prior for every model choice. Domingos assembles the working knowledge that follows: generalisation is the goal and training error is not a proxy for it, data usually beats cleverness, feature engineering is where effort pays, and simplicity does not by itself imply accuracy.
Why it matters for backtesting
If the goal is to understand why a strategy works, so that its failure can be anticipated, an interpretable model or an explicit explanation of a predictive model is needed for that decision. If the goal is the best prediction on the next period, accuracy on properly held-out data is the criterion that counts and interpretability is optional. An agent should ask which question is being asked before recommending anything. No free lunch supplies the accompanying caution: a model class that works on the user's instruments does so because its assumptions match those series, so a method that succeeded elsewhere carries no guarantee, and the comparison must be run on the data at hand. Compare candidates on a chronologically held-out sample, record the target and error measure, and check whether the simpler model remains adequate at the same decision horizon.
Source
Breiman, "Statistical Modeling: The Two Cultures", Statistical Science 16(3), 2001, 199-231; Wolpert, "The Lack of A Priori Distinctions Between Learning Algorithms", Neural Computation 8(7), 1996, 1341-1390; Domingos, "A Few Useful Things to Know About Machine Learning", Communications of the ACM 55(10), 2012, 78-87. Primary source