feat(stats): state the effect a sample size could not have detected

[jramos]

Why

A gate can certify a win, or enforce a regression floor, while its sample size was never capable of detecting the effect it claims to police. Neither the verdict nor the run's evidence said so.

From the new console line, on genuinely noisy data:

power: n=8, smallest detectable effect ≥0.928 (one-sided α=0.050, power=0.80);
       observed Δ=+0.012 is below it — this sample could not have shown an effect that small

What ships

min_detectable_effect_paired in evolution/core/stats.py, surfaced as power_diagnostics.json beside the decision plus that console line. Continuous regime only — see the withdrawal below.

α and sidedness are derived, not hardcoded. paired_bootstrap returns a two-sided interval at confidence and the gate consumes only its lower bound — a one-sided decision. Hardcoding two-sided 0.05 against the 0.90 default would inflate the figure by ~13% (not the ~19% the z-terms alone suggest: the power term is common to both). Pinned in a test.

The figure is labelled a lower bound, and that direction was verified. Against an exact noncentral-t computation the normal approximation is low by 11.3% at n=8 and 4.8% at n=16 — understating, which is the safe direction for a diagnostic about what a sample could not see.

ddof is real. The function takes the raw per-example differences and computes the spread itself. An earlier version recorded ddof without using it while the caller hardcoded stdev, which let the payload misreport its own provenance.

decision_rule is recorded, because some runs decide by means that never consult the interval (the closed-loop constraint discards it), and reporting an α as though it governed those would describe a rule that never ran.

Withdrawn before release: the paired-binary regime

It was implemented, reviewed, and pulled. Recorded because the reason is instructive:

• For paired binaries |p01 − p10| ≤ p01 + p10 is a hard algebraic bound, and (z+z)·√(p_disc/n) exceeds it whenever n·p_disc < 6.18 — i.e. across n≈8–20, this project's entire operating range. At n=16, p_disc=0.25 it reported 0.311 against a maximum possible 0.25.
• Both the docs example and a test I described as hand-computed pinned that impossible value. The hand check recomputed my own formula, so it could only ever catch arithmetic, never a wrong model. The bound check caught it immediately.
• Its is_lower_bound: True was wrong-signed: that property comes from the t-vs-z penalty on an estimated variance, which the binary case doesn't have. Against the Connor form it overstated by up to 60% at n=8. A wrong-signed honesty caveat is worse than none.
• The discordance feeding it came from continuous judge differences, almost never exactly equal, so the rate drifted to 1.0 and the number was 2.4865/√n restated.

Doing it properly needs the Connor form and real pass/fail counts, not thresholded score differences — the same rule already applied to the exact paired tests.

Diagnostics only — checked, not asserted

An earlier version compared evolve_code's decision payload against a pre-change golden. That golden is genuine, but it guards a gate this work never touches, so it proved something true and irrelevant. The skill/tool payloads can't be built without a full evolution run, so the check is now structural: the writer's values must be consumed by nothing but the console line. Mutation-verified — feeding one into growth_pass fails it.

The evolve_code golden is retained as an ordinary regression guard, relabelled honestly.

Verification

• 15 tests: known answers, the α derivation and the ~13% figure, the lower-bound labelling, the √n scaling relation, degenerate input, absence when nothing was scored, the sign-preserving console line, and the structural invariance check.
• Full non-slow suite: 1828 passed, ruff clean. No new dependency — quantiles from stdlib statistics.NormalDist.

Stacking

Based on the gate-integrity branch; GitHub retargets as those merge.

Rebuilt natively from statistics in NousResearch/hermes-agent-self-evolution#162. The exact-test half of the long-parked NousResearch/hermes-agent-self-evolution#136 remains deferred. No upstream diff applied.