stochastly
Start free trial

Library / Research method

A backtest is reported with the number of trials, or it is not reported

Which trials must accompany a reported winning backtest?

Report the candidate configurations, changes made after viewing results, data windows, costs and the rule that selected the final strategy. The trial count explains the search history; it does not automatically increase the threshold for a new lever or one sealed evaluation.

Report trials as search history, never an automatic penalty for added levers or a sealed batch.

The run records four searched and evaluated configurations; no selection gate was wired.
The run records four searched and evaluated configurations; no selection gate was wired.

Evidence map

AspectFinding
What it isA disclosure standard. A backtest result is a statistic computed on one sample after a search, and the search is part of the result.
The pitfall it addressesThe count is rarely withheld deliberately.
How to apply itRecord the trial count as the trials happen, and count a variant the moment it is evaluated, before any decision to keep it.

What it is

Reporting the performance without reporting how many configurations were examined gives the reader a number they cannot interpret, because the same figure means something entirely different after one attempt and after two hundred. The rule is therefore that a result travels with its trial count, the family of configurations that was searched, the period and universe used, and the decisions taken after seeing intermediate output.

The pitfall it addresses

It is lost, because no record was kept of the variants abandoned after a glance, the parameter values tried and forgotten, the filters added and removed, or the earlier version of the idea that was reshaped into the current one. Reconstructed afterwards, the count is systematically too low, and a correction applied to an understated count gives a false sense of rigour that is worse than no correction, since it carries the authority of a calculation. The second failure is scope: a result reported without its universe and period invites the reader to assume it generalises, and the researcher who omitted them usually chose them after seeing which worked.

How to apply it

Register the intended family before the search begins, so that its size is known in advance rather than measured by memory. When a result is written into a verdict, attach the count, the family, the period, the universe and any choice made after seeing output. When any of these is unavailable, state it explicitly and classify the result as unverifiable, a separate category from weak: a missing denominator is the absence of the evidence, which is more serious than a small defect in it. A result inherited from someone else carries their search, which is unknowable, so it is held to the stricter standard.

Source

Fabozzi & Lopez de Prado, "Being Honest in Backtest Reporting: A Template for Disclosing Multiple Tests", Journal of Portfolio Management 45(1), 2018, 141-147. Primary source

Related in the library