Validation checks

A test suite reports what passed. These validations also report what they could not check, and each one must demonstrate that it still detects its target failure.

A check must demonstrate that it can fail

A check that keeps passing and a check that cannot fail produce the same output. The difference appears when the behaviour breaks and the check still passes.

To expose that case, a validation builds the defect it was written to catch and confirms that the result fails. One export check once passed on an already broken rendering because it had been shaped around the broken text; replaying the defect exposed it.

Verdict / gate in Stochastly, Machine learning · cross-section, light theme. — The recorded ML run returns NO-GO and names the gates it failed.

How many validations can demonstrate their own failure

Figure 1. One square per validation, shaded when its failure can be replayed68 of 161 validations build a case that has to fail, then verify that it does. A passing check with no demonstrated failure may be unable to detect its target. 5 of them expose the failing case through an environment variable, so it can be replayed externally.

These failure cases target realistic defects. On the C++ export, filling not-yet-computed windows with zero in place of NaN makes five kernels fail, one of them by a single signal out of fifteen. A deliberately broken addition would demonstrate less: the validation can see the obvious.

5 of them are wired to an environment variable, which allows the failure to be replayed without editing the file. That number is low, and it is the one worth raising: a failure case that cannot be triggered externally remains short of a demonstration.

Multiverse pipeline replays in Stochastly, Full desk · 155 nodes, light theme. — Pipeline replays expose the outcomes of the alternative scenarios recorded by the run.

« I could not check » is a result, with its own exit code

Figure 2. Validations that can answer « I could not check »91 of 161 validations return a third exit code when a prerequisite is missing — for example, an unavailable engine or an unbuilt site. The other 70 can only pass or fail, so a pass may cover an empty measurement.

Some checks need something the machine does not have: a trading engine that lives outside the package, a compiler, or a built site. Forced into a binary answer, such a validation either accuses sound code or reports a pass over an empty measurement. Each validation now names the prerequisite it could not reach.

exit 0    PASSED       the property was measured, and it holds
exit 1    FAILED       the property was measured, and it fails
exit 2    NOT CHECKED   nothing was measured; a prerequisite was missing

A recorded failure case still falls short of proof

The figure above counts validations that contain a failure case. Whether each one still fails today is a separate question, answered by triggering them one by one. The count is therefore a floor on what has been exercised.

The same caution applies to coverage. Three export targets are checked without execution, since no interpreter for them runs on the build machine, and one target carries no numeric measurement. Those gaps are reported on the fidelity page, with the numbers that exist and the ones that do not.

Robustness grid in Stochastly, Machine learning · cross-section, light theme. — Measured robustness checks are plotted against their thresholds. Unmeasured checks remain explicitly labelled.