Look Before You Compute: Tukey's EDA and Anscombe's Quartet
Can the same summary statistics hide radically different trading data?
Plot the raw observations, time order and residuals before trusting a shared mean, correlation or regression coefficient. Tukey?s quartet shows that identical summaries can hide different shapes; for trading data, inspect gaps, outliers and regime segments explicitly.
Using the quartet as a strategy-audit analogy is an editorial inference.
Evidence map
| Aspect | Finding |
|---|---|
| What it is | Anscombe's four small datasets share the same summary statistics and the same fitted line yet describe four entirely different situations, built to refute the belief that "numerical calculations are exact, but graphs are rough". |
| Key result / formula | Each of Anscombe's sets has eleven (x, y) pairs with equal means, equal variances, the same correlation and the same least-squares line to the precision anyone would report. |
| Why it matters for backtesting | A backtest summary is a set of Anscombe numbers: mean return, Sharpe ratio, win rate, profit factor. |
What it is
Tukey's book turned the same conviction into a discipline: exploratory data analysis, done with plots and resistant summaries before any model is fitted, with confirmation reserved for fresh data.
Key result / formula
Plotted, the first is a linear relation with ordinary scatter; the second is a smooth curve, which a line summarises wrongly; the third is an exact line with a single outlier that drags the fitted slope; the fourth has all x values equal but one, so a single point determines the entire regression and the correlation is that point's doing. The statistics are identical, and the pictures alone show which case one is in. Tukey's contribution is the toolkit and the attitude: stem-and-leaf displays, box plots, resistant lines fitted through medians, residual plots examined for structure, re-expression (logs, roots) chosen from the shape of the data, and the insistence that exploration and confirmation are different acts on different data. His resistant statistics are designed so that one wild value cannot move them, the opposite of the mean and the least-squares slope, which are exactly the summaries a backtest report prints.
Why it matters for backtesting
Each of the four pictures has a strategy analogue. The outlier case is the equity curve made by one trade or one week; the single-point case is the strategy whose P&L is one episode, one market or one year; the curve case is a signal whose relation to forward returns is non-monotonic, so a linear rule captures the wrong part of it; the first case alone is what the summary implies. The exploration a user can do on bars is free: plot the equity curve, the distribution of trade returns, the return by year and by market, and the signal against the forward return by decile. The confirmatory test is the resistant one: remove the single largest trade and the single best month and recompute the verdict; if it flips, the summary described Anscombe's third or fourth set and the strategy has not been shown to exist (see [One extreme event early in the sample poisons expanding-window statistics]). Tukey's separation also names Stochastly's split: exploration is done on the training window, and confirmation belongs to bars the exploration did not touch.
Source
Anscombe, "Graphs in Statistical Analysis", American Statistician 27(1), 1973, 17-21; Tukey, Exploratory Data Analysis, Addison-Wesley, 1977. Primary source