Measurement protocol

A figure with no way to reproduce it is not a weak claim: it is an unfalsifiable one, which is a worse state than being wrong. The protocol below exists so that a reader can disagree with a number by running something.

The pattern is taken from a project that publishes its own testing figures: a count, the version and date it was taken at, and the definition of the unit in the same sentence. What makes such a line acceptable is the denominator written beside it.

The catalogue figures are generated from the desktop package. Pages hold no numeric literal of their own.

Each publishable number has a definition for its denominator and measurement. There are 15 governed figures; a value with no definition does not reach a page.

Measurements run before publication. The value shown must come from the object its label describes. A nearby total is a different object.

The dates differ, and that is the point of printing them. A single date on a page carrying five campaigns would suggest one run, and a figure measured three days earlier would inherit the freshness of the most recent one.

An automated freshness check rejects a generated measurement older than the product data it describes. This prevents the site from keeping an old count after the package changes.

Counting unique family/name identities gives 667 nodes, 175 of them marked roadmap. Counting the keys built by the enriched catalogue gives 688. The gap is measured, and its cause is unattributed.

Kernels, agent tools and nodes have each produced competing totals. A total is the result of a definition, so two counts of the same object diverge until one definition is written down. The site publishes the shipped catalogue and names the counting rule beside the figure.

A lower bound is rounded down to the ten before publication, so that a bound cannot drift upward as the catalogue grows.

A performance figure carries a denominator too, and it is the width of the search that produced it. That width is the effective number of trials that Romano-Wolf derives from the p-value of the maximum, and no deflated figure is returned for a run where it is missing. A Sharpe that came out of a wide sweep therefore arrives already discounted for the width of that sweep, and the corrections that do the discounting are listed with the papers they come from on the research index.

Statistical methods →

0    PASSED       the property was measured, and it holds
1    FAILED       the property was measured, and it fails
2    NOT CHECKED  nothing was measured; a prerequisite was missing

The third code prevents missing data from being reported as a pass. A result with no measurement is named separately.

Of the 161 validations measured on 2026-08-24, 68 carry a deliberately failing case. A recorded failure case is necessary but not sufficient: it still has to be triggered.

A validation also has to run the way its launcher runs it. The publishable checks run on synthetic bars in memory, with no disk and no port.

Static analysis reads 105 package modules and resolves declared hosts. 7 call sites build their URL while running, so they are counted without an invented name.

The same principle applies to the export campaign: the targets with no local interpreter are reported apart from the ones settled by execution, and the target with no compiler on this machine reports the gap by name and leaves the result unresolved.

The kernel count on the backtesting page comes from an interim validation environment. It is marked as interpolated because the shipped binary can settle that figure.

The protocol covers counts derived from the shipped catalogue and outputs of commands run on a named machine. It covers no performance figure, no benchmark timing and no hardware measurement. Those go up when a production build exists to measure.

Validity domain →