feat(stats): state the effect a sample size could not have detected

docs/data_models.md

@@ -734,6 +734,43 @@ Written by `evolution/code/evolve_code.py`. Unlike the GEPA variants (bootstrap
 
 `decision_signal` is `deterministic_test` (held-out-split model) or `oracle_match` (campaign/measurement model). `holdout` / `holdout_test` / `visible_test` belong to the held-out gate; the **oracle** variant instead carries `test` (the full fix-commit test file), `bug_tests` (list), `oracle_failure_count`, `guards.bug_tests_passed`, and a `guards.oracle_match` block (`new_vs_oracle`, `oracle_failure_count`) — there `guards.floor` is often `null` because oracle-match over the full file *is* the regression check. `full_suite` is the optional `--benchmark-cmd` block (and may add `downgraded_from: "deploy"` if it demoted a pass).
 
+### `power_diagnostics.json`
+
+Written beside `gate_decision.json` by the skill and tool evolvers
+(`evolution/core/power_report.py`). **Diagnostics only** — no gate reads them, and a
+structural test pins that the values are consumed by nothing but the console line.
+
+```json
+{
+  "n_examples": 10,
+  "observed_mean_difference": 0.08,
+  "decision_rule": null,
+  "alpha_describes": "the lower bound of the paired bootstrap interval",
+  "continuous": {"mde": 0.062, "n": 10, "sd_diff": 0.079, "alpha_one_sided": 0.05,
+                 "power": 0.8, "ddof": 1, "method": "normal-approximation",
+                 "is_lower_bound": true}
+}
+```
+
+`mde` is the smallest effect this sample size could reliably detect: when the observed
+difference falls below it, a passing gate is not evidence of a win. `alpha_one_sided` is
+derived from the same `confidence` the bootstrap uses, because the gate consumes only the
+interval's lower bound — and `decision_rule` records when the run actually decided by some
+other means (the closed-loop constraint discards the interval entirely), so the alpha is
+not read as governing a rule that never ran. `is_lower_bound` is always true: this uses the
+normal approximation, while the exact noncentral-t value is larger — by about 11% at n=8
+and 5% at n=16 — so the figure understates, which is the safe direction for a diagnostic
+about what a sample could not see.
+
+Continuous regime only. A paired-binary companion was written and withdrawn before release:
+`|p01 - p10| <= p01 + p10` is a hard algebraic bound, and the normal approximation violates
+it whenever `n * discordance < 6.18`, which covers this project's entire operating range.
+Doing it properly needs the Connor form and real pass/fail counts rather than differences in
+continuous judge scores.
+
+The file is **absent** on runs that abort before scoring, which means "not computed" rather
+than "nothing to detect".
+
 ### `repair_trace.json`
 
 Per-round repair record for human review (`evolution/code/trace.py`). No per-hunk attribution.

docs/upstream_pr_triage.md

@@ -405,6 +405,55 @@ and we are 0 commits behind it, so waiting accrues no merge risk.
   **Retention lesson:** per-seed exit codes are a few bytes and would have made this a grep instead
   of an inference; the repaired sources are gone for good.
 
+- **2026-08-31** — **Minimum detectable effect** (from #162's statistics, rebuilt) **added**. A gate
+  could certify a win, or enforce a regression floor, while its sample size was never capable of
+  detecting the effect it claimed to police — the verdict said nothing about that, and neither did
+  the run's evidence. One pure function in `stats.py` now states it, for the continuous regime
+  (judge scores), surfaced as a `power_diagnostics.json` beside the decision plus a console line.
+  A paired-binary companion was written and withdrawn before release (below).
+  Three choices worth recording. α and sidedness are **derived from the bootstrap's own
+  `confidence`** rather than hardcoded, because the gate consumes only the interval's lower bound —
+  a one-sided decision; hardcoding a two-sided 0.05 against the default 0.90 interval would inflate
+  the figure by about 13% (the z-terms differ by 19%, but the power term is common to both). The
+  withdrawn binary version had been parameterised by **discordance rate, not marginal pass rate** —
+  paired power
+  depends on how often the arms disagree, and closed-loop pass counts are strongly correlated by
+  construction, exactly where a marginal model reads optimistic. And the figure claims **no bound**: these use the normal approximation while the correct quantile at n of 8-20 comes
+  from a noncentral t and is larger, and understating is the unsafe direction for a diagnostic whose
+  purpose is admitting what a sample cannot see.
+  Diagnostics only — nothing reads them, and that is checked rather than asserted: a structural test
+  requires the diagnostic's values to reach nothing but the console line, mutation-verified by
+  feeding one into a gate variable and watching it fail.
+  Review then **withdrew half of it before release**. The paired-binary companion returned values
+  above the algebraic maximum: `|p01 - p10| <= p01 + p10` bounds the effect, and the normal
+  approximation violates that bound whenever `n * discordance < 6.18` — this project's entire
+  operating range. Both the worked example in the docs and a test described as hand-verified pinned
+  an impossible number, because the hand check recomputed the same wrong formula instead of checking
+  the bound. Its lower-bound flag was wrong-signed as well (overstating by ~14% at n=8 rather
+  than understating), and the discordance feeding it came from continuous judge differences, which
+  are almost never exactly equal — so the rate drifted to 1.0 and the figure was `2.4865/√n`
+  restated, carrying nothing about the run. Deferred with those reasons; doing it properly needs the
+  Connor form and real pass/fail counts. The continuous half was verified against an exact
+  noncentral-t computation, including the direction of its lower-bound claim (low by ~11% at n=8,
+  ~5% at n=16).
+  A second review round caught the sharper version of the same mistake in the half that shipped.
+  The lower-bound direction had been verified against an exact paired **t-test** — but the gate does
+  not run one. `paired_bootstrap` returns a *percentile* interval whose spread is the divisor-n
+  resample sd with no t-correction, so rejecting on its lower bound is equivalent to requiring
+  `t > z * sqrt((n-1)/n)`: 1.5386 at n=8, against a nominal 1.6449. That rule is **anti-conservative**
+  (real one-sided error ≈0.08 where 0.05 is claimed), so its true detectable effect is *below* the
+  reported figure and `is_lower_bound: True` was wrong-signed for the decision it sat beside — the
+  very defect that got the binary regime withdrawn, surviving in the continuous half. The arithmetic
+  had been right and the conclusion wrong, because the number was checked against a test this
+  codebase never runs. The diagnostic now models the gate's own rule (reporting the effective
+  critical multiplier) and claims no bound in either direction.
+  Two further corrections from the same review: `ddof` was recorded but never used while the caller
+  hardcoded the spread — a knob that looked like a parameter and changed nothing, letting the
+  payload misreport its own provenance — so the function now takes the raw differences; and the
+  first invariance test guarded the *code* evolver's payload, which this work never touches, proving
+  something true and irrelevant. Exact paired tests stay deferred. 15 tests; full non-slow suite
+  green (1828 passed), ruff clean.
+
 ## Action items (open)
 
 Disposition lens: against our diverged tree, these are "rebuild the idea/mechanism
@@ -422,7 +471,7 @@ ourselves," never "merge the PR" — we do not apply upstream diffs. Our-code an
 | #142 (partial) | **Symlink-aware skill resolver** — `find_skill` traversal follows symlinked skill directories | `Path.rglob("SKILL.md")` doesn't descend into symlinked dirs on Python <3.13; a Hermes layout that symlinks user-installed skills into the framework tree would silently resolve "not found." Real latent bug in a path we own. | `evolution/core/skill_sources.py` (`HermesSkillSource`, was 3 `rglob("SKILL.md")` sites) | **DONE** — replaced the three `rglob` sites with one cycle-safe `_iter_skill_files` helper (`os.walk(followlinks=True)` + `(st_dev, st_ino)` visited-set + sorted deterministic order); only `HermesSkillSource` touched (flat ClaudeCode/LocalDir sources already follow symlinks via `is_file()`). 9 new symlink tests, full non-slow suite green (1763) — see review log, 2026-07-06 | ✅ |
 | #149 (+ #26) | **Rank importer pre-filter candidates by relevance** before the LLM-scoring cap — graded score replacing the boolean predicate, strongest-first ordering | `RelevanceFilter` qualifies candidates with a boolean predicate and then truncates at `max_examples * 3` in source-then-import order, so the strongest matches past the cap never reach the LLM scorer. The scoring loop's early break means order decides the output set on every run, not only on overflow. Closes the #26 recall follow-up. | `evolution/core/external_importers.py` (`_is_relevant_to_skill`, `RelevanceFilter.filter_and_score`) | **DONE** — `_relevance_score` returns a tiered tuple `(name_match, name_words, keyword_overlap)`; `_is_relevant_to_skill` is now `any(...)` over it, so the qualifying set is unchanged; candidates sort strongest-first (stable, ties keep import order) ahead of both caps — see review log, 2026-08-31 | ✅ |
 | #162 (partial) | **Confine code-evolution test execution** and record the containment posture | `WorktreeEnv.run_test` executes pytest against an LLM-modified worktree through a bare subprocess, while the agent runner refuses to run unconfined at all — an asymmetry in our own doctrine. The adversarial gaming harness runs through the same path. | `evolution/code/worktree.py` (`run_test`), `evolution/validation/claude_runner.py` (the existing profile), `evolution/code/gate.py` (failure parsing) | **DONE** — shared `evolution/core/sandbox.py` (profile + availability + `wrap_argv`); `run_test` confines writes to the run root, records the posture in `repair_trace.json`, and raises rather than returning a non-pytest exit code from a confined run; `--require-sandbox` on all three LLM-loop entry points — see review log, 2026-08-31 | ✅ |
-| #162 (partial), #136 | **Minimum detectable effect** as a gate-adjacent diagnostic | A gate can certify a win, or enforce a regression floor, without ever stating that its sample size could not detect the effect it claims to police. Absorbs the parked exact-test item. | `evolution/core/stats.py` (currently `paired_bootstrap` only) | **REBUILD** — MDE for the continuous and paired-binary regimes, diagnostic only, deploy decisions provably unchanged; exact paired tests deferred rather than shipped with unpinned conventions | ⬜ |
+| #162 (partial), #136 | **Minimum detectable effect** as a gate-adjacent diagnostic | A gate can certify a win, or enforce a regression floor, without ever stating that its sample size could not detect the effect it claims to police. Absorbs the parked exact-test item. | `evolution/core/stats.py` (currently `paired_bootstrap` only) | **DONE (continuous only)** — `min_detectable_effect_paired` in `stats.py`, surfaced as `power_diagnostics.json` beside the decision; α and sidedness derived from the bootstrap's own confidence, the figure labelled a lower bound (verified against exact noncentral t). The paired-binary companion was written and **withdrawn** — it violated `|p01-p10| <= p01+p10` across our whole operating range; it needs the Connor form and real pass/fail counts. Exact paired tests still deferred — see review log, 2026-08-31 | ✅ |
 | #154, #179 | **Importer and dataset-builder hardening** — malformed JSON raises our own error; non-UTF8 session files are skipped instead of crashing the importer | Two sites extract a JSON substring and then parse it unguarded, so a bracketed-but-malformed payload escapes as a raw decode error. Separately, `UnicodeDecodeError` is uncaught where legacy session files and the Claude Code history log are decoded — and the history log is the likelier of the two to be mixed-encoding. | `evolution/core/dataset_builder.py`, `evolution/core/external_importers.py` | **REBUILD** — guard both parse sites (including a shape check, since valid JSON of the wrong type escapes just as badly) and widen both decode guards; no new dependency, declining the proposed JSON-repair library | ⬜ |
 | — (found in review) | **Authoritative failure sets** — a pytest run that could not answer must not be scored as "nothing failed" | `failing_tests` discarded the exit code, and the failure parser returns an empty set on unrecognised output, so a timed-out, killed or uncollectable run certified a wrong repair as `correct` through the oracle gate. Demonstrated against the real gate. | `evolution/code/worktree.py` (`failing_tests`), `evolution/code/harvest.py` (`_failures`), `evolution/code/gate.py` (the parser) | **DONE** — the seam refuses a run that produced **no failure evidence and no authoritative exit**; a *stricter* guard added to both regression floors and the held-out check — a diff needs a complete failure set, not merely some evidence, so those demand an authoritative exit where the seam accepts any run that named a failure; `harvest._failures` delegates instead of re-parsing; a distinct error type keeps the ledger honest, and an inconclusive run while scoring counts as a failed seed rather than a dropped organism — see review log, 2026-08-31 | ✅ |
 | #174 (partial) | **MIPROv2 fallback receives the held-out valset** | The fallback optimizer compiles without `valset` while the primary path passes it, so a fallback run loses its held-out set for internal candidate selection. Narrow: deploy integrity is unaffected, since the deploy gate runs its own held-out behavioral validation downstream. | `evolution/skills/evolve_skill.py` (`_default_mipro_runner`) | **REBUILD** — pass the existing named val split through, guarding the empty case (the optimizer rejects a non-None empty valset). Threading `num_trials` stays excluded: the optimizer treats it as mutually exclusive with the `auto` preset. | ⬜ |

evolution/core/power_report.py

@@ -0,0 +1,119 @@
+"""Write the power diagnostic that sits beside a gate decision.
+
+Deliberately a separate artifact from ``gate_decision.json``. This number is
+context for reading a verdict, never an input to one, and keeping it out of the
+decision payload is what makes that claim testable rather than asserted.
+
+Continuous regime only. A paired-binary companion was written and withdrawn: it
+emitted values above the algebraic maximum ``|p01 - p10| <= p01 + p10`` across
+this project's entire operating range, and the discordance it would have been fed
+here — per-example judge differences that are almost never exactly equal — is not
+the pass/fail disagreement such a model is about.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Optional
+
+from evolution.core.stats import min_detectable_effect_paired
+
+_FILENAME = "power_diagnostics.json"
+
+
+def build_power_diagnostics(
+    baseline_scores: list[float],
+    evolved_scores: list[float],
+    *,
+    confidence: float = 0.90,
+    power: float = 0.80,
+    decision_rule: Optional[str] = None,
+) -> dict:
+    """What effect this sample size could, and could not, have detected.
+
+    ``decision_rule`` is recorded because the reported alpha describes the
+    *interval* rule. Some runs decide by other means — a point estimate against
+    zero, or the closed-loop constraint, which discards the interval entirely —
+    and reporting an alpha as though it governed those would describe a rule that
+    never ran.
+    """
+    # Checked before the emptiness short-circuit below, or a mismatched pair with
+    # an empty baseline would slip through while its mirror raises.
+    if len(baseline_scores) != len(evolved_scores):
+        raise ValueError(
+            f"power diagnostics need paired arrays of equal length; got "
+            f"{len(baseline_scores)} baseline vs {len(evolved_scores)} evolved"
+        )
+    n = len(baseline_scores)
+    diffs = [e - b for b, e in zip(baseline_scores, evolved_scores)]
+    out: dict = {
+        "n_examples": n,
+        "observed_mean_difference": (sum(diffs) / n) if n else 0.0,
+        "decision_rule": decision_rule,
+        "alpha_describes": "the lower bound of the paired bootstrap interval",
+    }
+    if n > 1:
+        cont = min_detectable_effect_paired(diffs, confidence=confidence, power=power)
+        cont["alpha_one_sided"] = round(cont["alpha_one_sided"], 6)
+        out["continuous"] = cont
+    return out
+
+
+def write_power_diagnostics(
+    output_dir: Optional[Path],
+    baseline_scores: list[float],
+    evolved_scores: list[float],
+    *,
+    confidence: float = 0.90,
+    power: float = 0.80,
+    decision_rule: Optional[str] = None,
+) -> tuple[Optional[Path], Optional[dict]]:
+    """Write the diagnostic beside the run's other artifacts, if there is a dir.
+
+    Returns ``(path, payload)``; both are None when there is nothing to write. A
+    missing file means "not computed" — runs that abort before scoring never
+    reach here — and never "nothing to detect".
+    """
+    if output_dir is None or not baseline_scores:
+        return None, None
+    payload = build_power_diagnostics(
+        baseline_scores, evolved_scores, confidence=confidence, power=power,
+        decision_rule=decision_rule,
+    )
+    output_dir.mkdir(parents=True, exist_ok=True)
+    path = output_dir / _FILENAME
+    path.write_text(json.dumps(payload, indent=2) + "\n")
+    return path, payload
+
+
+def format_power_line(payload: dict) -> str:
+    """One console line: what the run could have seen, next to what it saw.
+
+    Keeps the sign of the observed difference. For a gate that only ever
+    certifies improvements, a regression reported as a bare magnitude "above" the
+    detectable effect reads as a well-powered win — the sign is the one bit that
+    must not be dropped.
+    """
+    cont = payload.get("continuous")
+    if not cont:
+        return "  power: too few examples to state a detectable effect"
+    observed = payload.get("observed_mean_difference", 0.0)
+    if cont["mde"] == 0.0 and observed == 0.0:
+        # Identical arms: no variation to power a test on, and no effect to detect.
+        # Strict "<" would render this as an effect *above* the detection floor.
+        return (
+            f"  power: n={cont['n']}, arms are identical — no variation between them, "
+            "so there is nothing to detect and nothing detected"
+        )
+    if abs(observed) <= cont["mde"]:
+        verdict = "below it — this sample could not have shown an effect that small"
+    elif observed < 0:
+        verdict = "above it, but negative — a detectable regression"
+    else:
+        verdict = "above it"
+    return (
+        f"  power: n={cont['n']}, smallest detectable effect "
+        f"≥{cont['mde']:.3f} (one-sided α={cont['alpha_one_sided']:.3f}, "
+        f"power={cont['power']:.2f}); observed Δ={observed:+.3f} is {verdict}"
+    )