Known limits of the research

By Mahé Chadeffaud, founder of Stochastly

Each entry names the perimeter it touches, the direction of its effect, and the control available in the build today.

An entry carries a mechanism, not an apology

An entry earns its place by naming three things: the quantity or the decision it touches, the direction in which it moves a result, and what the build offers against it today. A limit stated without those is an impression, and it stays out.

Each entry states a condition, and that condition is what makes a number produced by this engine interpretable. A reader who skips the entry reads the number wider than it was measured.

What the file contains, and what it leaves out

Survivorship

Scope: equity universes built from a currently-listed index
Effect: may overstate performance, since delisted names are absent from the file
Control: load a universe file that carries delisted tickers

Point-in-time on prices

Scope: price series; alternative series are covered by the next entry
Effect: a revised price can flatter a rule fitted on the revised value
Control: alternative series can carry an explicit vintage field

Point-in-time on alternative series

Scope: macro and alternative series whose publication date is missing from the source
Effect: an observation can be read earlier than it was published, which flatters a rule fitted on it
Control: the app states that it cannot promise point-in-time for that series, and one alignment stage controls the date shift

A bar column read as an external series

Scope: alternative-data nodes wired onto an OHLCV file
Effect: a node could bind to CLOSE or VOLUME as if it were an external series, and 0 checks raised
Control: the loader keeps observation dates and knowledge dates in separate fields

A missing column in an engine kernel

Scope: a V9 kernel reading a column the bar file does not carry
Effect: the run proceeds on a fabricated constant, so the figure is not a measurement of your data
Control: the result carries a data warning that marks the figure as not measured

What is assumed about getting filled

Capacity and market impact

Scope: sizing and fills on any run
Effect: fills are assumed executable at the size you set
Control: cost nodes apply the spread, commission and slippage you declare

Undeclared costs

Scope: a graph wiring no cost node
Effect: the run is gross, and gross overstates what the account would have kept
Control: the gross label travels with the run into the verdict and the exported paper

What the gate proves, and where it stops

Boundary margin of the causality gate

Scope: backends holding a position across the truncation cut
Effect: a dependency confined to the boundary window passes the production gate — a backend reading one bar ahead is measured at exactly zero disagreement
Control: a second validation replays the same test with the margin set to zero

What the causality validation covers

Scope: the shipped kernels, on synthetic bars, in memory
Effect: it establishes the causality of a kernel, and covers 0 properties of a real data set
Control: the validation states its perimeter and reports not checked when the engine is absent

What the corrections settle, and what stays open

Multiplicity filters, and stops there

Scope: any run scoring more than one configuration
Effect: a configuration surviving the correction is not thereby deployable
Control: the multiplicity result states this scope; deployment is judged separately

Too few independent observations

Scope: runs with fewer than 30 effective observations, fewer than 30 common bars, or fewer than 2 finished configurations
Effect: a number would be produced from a sample that cannot support it
Control: the result is returned as insufficient with the figure that failed

A ceiling on what is corrected

Scope: runs above 400 configurations, and validation matrices above 3000 bars
Effect: beyond the ceiling the correction runs on a sample, and the matrix keeps the most recent bars
Control: the sampling is declared in the report rather than truncated in silence

False-discovery control assumes a dependence structure

Scope: the Benjamini-Hochberg procedure over correlated configurations
Effect: two configurations long and short on the same signal can be negatively correlated, which BH does not cover
Control: BY is emitted beside BH so the gap between them stays visible

The overfitting probability weighs in no automatic score

Scope: PBO by CSCV
Effect: it is biased high under a hundred configurations, and splitting the same grid across two runs changes the penalty without the data moving
Control: it is excluded from the score and remains available as a node to wire

Purging is applied on the out-of-sample side

Scope: CSCV block boundaries
Effect: without the embargo the overfitting probability comes out optimistic — measured 0.306 against 0.357 at a 120-bar overlap
Control: the embargo is applied and the report states that the in-sample side is not purged

Two different outcomes, and they were written as one

A kernel the translator does not cover

Scope: graphs built on a kernel outside the 18 translated ones
Effect: the export is refused by name and no file is written
Control: Check the exact supported translations in the export kernel reference.

A translation below the coverage threshold

Scope: a translated kernel whose exported rule flags fewer of the engine's entries than the threshold
Effect: the file IS written below 0.8; the result carries verified:false and the measured coverage
Control: the reservation travels in the generated file header

What the coverage number measures

Scope: the compiled and transcribed targets
Effect: it is a recall on the engine's entries with a two-bar tolerance, and extra signals carry no penalty, so it states no equivalence
Control: the exported file states that the measurement used synthetic paths

Distance between a translated file and the run

Scope: the transcribed targets, measured over 1000 draws of 3000 bars
Effect: the worst target disagrees on 16.9 % of triggered bars (pine_avant)
Control: the campaign, its interval and its registry are published under export fidelity

What the placebo test can and cannot separate

The sizing placebo tests alignment, not stationarity

Scope: a modulator applied to position size
Effect: a modulator that reads ahead aligns perfectly and passes with the best score
Control: both limits are stated; the null group is enumerated when small enough, sampled and declared otherwise

Stationarity tests are informational

Scope: ADF, KPSS, Jarque-Bera, Engle-Granger, Johansen
Effect: no threshold is wired to them, so 0 configurations are excluded on their result
Control: p-values and critical values are returned for you to read; KPSS states that same-sample parameter estimation is conservative

What a published count is, and is not

A declared status is not a proof

Scope: the runnable-node count
Effect: a status declared in the catalogue sets a ceiling on a result, and does not establish one
Control: the published count includes checks that were actually executed

A recorded control is not an executed failure

Scope: the count of validations carrying a deliberately failing case
Effect: the failing case has to be triggered before it demonstrates detection
Control: the limit is written into the measurement itself

A result has a market regime

Splitting the recorded result by regime reveals where the aggregate can hide different behaviour. This is a description of this sample, not proof that the same regimes or returns will recur.

Regime split in Stochastly, Full desk · 155 nodes, light theme. — The recorded result separated into calm and volatile observations.

The tails can depart from the reference

A tail quantile plot compares the recorded return distribution with its reference. Departures matter when interpreting risk estimates: the reference curve is an assumption, not a guarantee about extreme losses.

QQ tail plot in Stochastly, Full desk · 155 nodes, light theme. — Empirical return quantiles against a normal reference; deviations in the tails remain visible.