Machine-learning validation

Three things move when a fitted model enters the graph. Model capacity meets a return series whose predictable share is small, so an in-sample fit absorbs the noise it was given. A label built over a horizon spans a split boundary, so a training row can carry the outcome window of a test row.

The third change is arithmetic. Features, hyperparameters and fold geometry all become choices, and each of them multiplies the count of specifications tried before a winner is picked. A selection price computed on the strategy grid alone understates that count.

The result of the three together is a curve that reads well in sample and reproduces poorly out of it. The readings below are the ones the shipped nodes attach to a model, so the three failures stay visible where they apply.

The machine-learning family ships 38 nodes, of which 24 are validated by executable checks, and 14 carry a declared roadmap reason. The regime family ships 18 nodes with 7 validated, and the validation family ships 47 with 41 validated.

One node of the family produces a P&L series. A cross-sectional model ranks the loaded universe each bar, takes the top of the ranking, adds the bottom of it when the declared direction is long-short, and is routed to a walk-forward backend that returns a curve. The rest of the family reads the panel a sweep already produced, and each of those returns a figure in place of a curve.

Among the reading nodes, the ones that fit an estimator cover gradient boosting, a random forest, an ElasticNet paired with a logistic classifier, a multilayer perceptron, an estimator you declare yourself, and a stacking ensemble. Beyond them the family covers a benchmark suite, a feature ablation gated by a predictive-accuracy test, conformal intervals, an out-of-distribution distance, an information-bottleneck selector, and two regime routers.

The nodes carrying a roadmap reason report it in place of a figure, and the reason names the missing input. A hyperparameter search reports the absent package it would need, and an execution agent reports that its decision variable, the placement inside a bar, is not present in the panel used by the calculation.

For the reading nodes of the family, the training set is the run itself. A sweep carries a matrix with one row per bar and one column per configuration tried, holding the per-bar P&L of each. The module turns that panel into a supervised problem: the features of a row are what was known about a column before a bar, and the label is that column's P&L on that bar.

The features are lagged values, rolling means over two spans, a rolling volatility, a share of winning bars, a panel-wide factor and a spread to it. No price series, order book or external feed reaches the module, so a figure it returns is computed on data the run produced.

A degenerate panel is refused with the degeneracy named. Fewer than three columns with non-zero variance, or off-diagonal correlations above 0.999 throughout, both stop the node: a model fitted on either would return an exact figure computed on nothing that varies.

The scope of the reading rides with it. A model that ranks those columns describes an arbiter over the configurations that run submitted. A complete system would also have had to pick the search space while walking forward, and that part is outside the panel.

Each feature passes through one of two constructions: a shift by a fixed number of bars, or a window closed on the left that aggregates the span from t minus w to t minus one. A row at bar t is built from bars up to t minus one.

The test covering this is destructive. The last bars of the panel are rewritten with other values, and the assertion holds when no earlier feature value has moved. A test that inspected the construction code alone would pass on a shift written the wrong way round.

At the backend seam a separate probe truncates future bars and compares the past predictions against the untruncated run. A backend whose past output moves under that truncation is refused before the sweep runs, so a model fitted on the whole series is stopped by the contract.

In the reading nodes of the family, test blocks start after a warm-up and march forward to the end of the history. The training set of a block is what exists before it, minus one bar of purge, so a split is anchored in time. A shuffled K-fold over the same panel trains on the future and returns an IC that no forward data reproduces.

The fold count comes from the node when the node carries one. Without it the count is derived from the history length, bounded, and its provenance is written into the report as a derived value. The seed follows the same rule, so a second run of the same graph returns the same output.

The geometry travels with the figures. The report carries the fold count and its origin, the seed and its origin, the test bar range of each fold, the training and test row counts of each fold, and the per-fold IC beside the pooled one.

The regime routers use the forward recursion alone. A smoothed state at bar t would be computed with the bars that follow t, and a router built on that state would read ahead while returning a plausible-looking series.

A reading node fitting a single estimator publishes a second IC beside the first. The same estimator, the same folds and the same features are replayed against a target permuted inside each bar, so the temporal and cross-sectional structure survives while the link between features and label is removed.

That second figure is the zero of the scale for the first. An IC that holds up under the permutation is measuring a path from label to features that survived the split, and a comparison against a fixed threshold would have missed it.

The benchmark suite applies the same treatment to its own metrics. IC, rank-IC, ICIR and a top-of-ranking gain are recomputed on shuffled predictions, so a metric that reports the shape of the panel becomes distinguishable from one that reports the fit.

The ablation removes a group of features and tests whether the loss differential moved. The differential is aggregated inside a bar before the test, because the columns of one bar share the same market shock; counting them as independent observations divides the standard error by about the square root of the column count.

The stacking node carries a control of its own shape. The same combination is replayed over three copies of one base estimator, so the gain the diversity of the bases produces stays separable from the gain the combination step produces on any set of inputs.

The conformal node splits each training prefix into a fitting part and a later calibration part, and reports the coverage observed at each target level with the mean interval width beside it. The calibration set is disjoint in time from the fit, and the finite-sample quantile is used in place of the bare empirical one.

The exchangeability assumption is stated in the output as an assumption. It holds approximately on a financial series, so the coverage published is the measured one and the theoretical guarantee stays an argument attached to it.

The out-of-distribution node measures the Mahalanobis distance of test features to the training mean and covariance, with the threshold set at the 95th percentile of training distances. Under stationarity the share of test points above that threshold is 0.05, and that value is the built-in control for the reading.

Neither node removes a configuration from the run. A gate that stopped trading outside the training distribution is a product decision, and it is left to the person wiring the graph.

Beside the family readings, the data family ships 19 nodes with 10 validated, and the validation nodes read the finished run at its own level. The combinatorial purged node exposes a split count, a test-group count and an embargo share, and the result carries the split count, the smallest and largest test and training sizes, and the bars each split gives up to purge and embargo.

The unit translation between node and engine is written into the result. A value posted on the embargo share is converted to a fraction before it reaches the engine, and a setting the engine signature does not accept is named in the result as unapplied. Without that sentence, a rename on the engine side would leave a live setting inert.

The overlap check counts train-and-test index intersections across the purged splits. The expected value is stated as 0 in the output beside the count, so a reading of 0 is distinguishable from a check that returned an empty result.

A separate walk-forward read marches anchored windows through the run, re-selecting the best configuration on prior bars and scoring that pick on the next unseen span. Its report carries the warm-up length, the window length, the window count, the dated bounds of each span, and the chained curve those picks would have traded.

Two nodes of the data family, one for purge and one for the post-test gap, reach the computation: their bar counts shrink the in-sample and out-of-sample windows that the wired gates read, and a test covers the change in selection an embargo of 400 bars produces. The values applied are absent from the fields of the run result, so a result read later does not state whether either node was set.

Two consequences follow from that absence. The family module applies a fixed purge of one bar to its own folds, independent of the purge node. And the cross-sectional model node exposes an estimator, a horizon, a fold length and a top share, with the embargo defaulting to the horizon and absent from its report.

The combinatorial block is the one that reads back: the embargo it computes travels into the result, next to the count of splits it left usable and the count it started from. Announcing the embargo without those two counts would hide what it cost.

The condition for this part to describe the remaining two as readable is a field in the run result carrying the purge and the embargo actually applied, at each level that applies one, with a test that fails when a declared value is missing from the result.

A production package contains the strategy engine; the build fails if that source is missing. The separate non-deliverable demo configuration can use a labelled generic cross-sectional fallback. Its curve is not evidence for the requested strategy.

No reading described above is a statement about future performance, and the harness carries no signal to trade. A result concerns the series that was submitted.