Explainability is a question about structure, not about magnitude - #144
Draft
amburger66 wants to merge 2 commits into
Draft
Explainability is a question about structure, not about magnitude#144amburger66 wants to merge 2 commits into
amburger66 wants to merge 2 commits into
Conversation
amburger66
force-pushed
the
domino-penalty-aware-explainability
branch
from
August 20, 2026 20:05
7dd99a2 to
8d0d522
Compare
amburger66
force-pushed
the
domino-penalty-aware-explainability
branch
from
August 20, 2026 21:49
7da1120 to
49f7e4d
Compare
The trimmer drops a recording no parameter value can explain, so a chaotic replay cannot drag the fit to a compromise nobody supports. It decides with an RMS bar: trim_rms_factor * noise_sigma = 0.1. Under interval scoring that bar is applied to two residuals that are nothing alike. A MISSING-CASCADE PENALTY says the twin failed to reproduce WHICH dominoes fall. A timing term says both sides saw a domino fall, at different moments. On run_20260820_141450 one penalty was 2884 in SSE while the entire timing disagreement was 5.3 -- so the bar is really a penalty detector with a wildly miscalibrated tail, and it rejects structurally perfect data: per-trajectory best RMS ['inf', 'inf', '1.21'] vs threshold 0.1000 NO trajectory is explainable at any candidate params That 1.21 is a theta at which the twin reproduces ALL FOUR falls with zero penalties -- the same theta the sweep reports as explaining the data 1084x better than baseline (0.2402 -> 5.321 against 5769 at the agent's belief). The fit refused to run on the best explanation it had, because the residual timing error the fit exists to REDUCE was used as grounds for refusing to fit. That is circular. So ask the question that was meant: did any candidate reproduce the cascade? Three outcomes rather than one number, and the middle one is the whole point: nothing measured at all -> not explainable (nothing to explain) any missing-cascade penalty -> not explainable (wrong dominoes fall) penalties absent, timing only -> EXPLAINABLE, and the fit does its job The first row is not padding. A cascade-free segment emits no penalties because it emits nothing, and reading "no penalties" as success there would reinstate exactly the inversion that made an empty residual vector an RMS of 0.0 -- vacuous segments outranking real ones. Counted from the id sets (a penalty IS a domino one side has and the other does not, which is the symmetric difference), never by recognising penalty-sized numbers in the output: magnitude-sniffing works until a real disagreement lands on the penalty value. THE PER-STEP OBJECTIVE IS UNTOUCHED. There the residuals really are a dimensionless fraction of typical motion, compute_residual_scaling put them there, and 0.1 means what it says -- so the RMS bar still decides, gated on score_observed_only. Scope: reproduced_cascade asks whether ANY candidate on the grid reproduced the cascade, not whether the RMS-best one did. Those differ -- a theta that completes the chain but mistimes it can score worse than one that stalls it early with fewer terms -- and "could this model ever produce this outcome" is a question about the whole grid. This IS more permissive than before: a recording where the twin gets the right dominoes falling but badly wrong timing is now kept and drives the fit. That is the data the fit should use, but it is a real behaviour change and not a refactor. min_explainable_fits gains a third return and the explainability cache a third element; the stub in test_trimming_uses_rms_cache moves with them. The new test fails under a revert of the structural switch, and exactly that one test fails -- it passes its own SysIdConfig rather than touching CFG, so interval scoring does not leak into the rest of the module.
…ract The explainability cache gained a third element (whether any candidate reproduced the segment's cascade) but two DECLARATIONS of its type did not follow: the approach's own attribute, and SynthesisBackend, the structural protocol the synthesis tools are typed against. A protocol mismatch does not surface where the attribute is declared -- it surfaces as the whole approach failing to satisfy the protocol at the call sites that pass it, which is why the error named create_predicate_synthesis_tools and not the cache. Caught by CI, not locally: I ran mypy over predicators/code_sim_learning/ and the failure was three directories away. CI runs `mypy . --config-file mypy.ini` over 721 files, and so does this commit's verification.
amburger66
force-pushed
the
domino-penalty-aware-explainability
branch
from
August 21, 2026 18:27
49f7e4d to
37d67c6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #143 — review that one first; this diff is only the last commit.
The problem
The trimmer drops a recording no parameter value can explain, so a chaotic replay cannot drag the fit to a compromise nobody supports. It decides with an RMS bar:
trim_rms_factor * noise_sigma = 0.1.Under interval scoring that bar is applied to two residuals that are nothing alike:
On
run_20260820_141450one penalty was 2884 in SSE while the entire timing disagreement was 5.3. So the bar is really a penalty detector with a wildly miscalibrated tail, and it rejects structurally perfect data:That
1.21is a theta at which the twin reproduces all four falls with zero penalties — the same theta the sweep reports as explaining the data 1084x better than baseline (0.2402 -> 5.321against5769at the agent's belief). The fit refused to run on the best explanation it had, because the timing error it exists to reduce was used as grounds for refusing to fit.The change
Three outcomes rather than one number:
The first row is not padding: a cascade-free segment emits no penalties because it emits nothing, and reading that as success would reinstate the inversion that made an empty residual vector an RMS of 0.0.
Penalties are counted from the id sets — a penalty is a domino one side has and the other does not, i.e. the symmetric difference — never by recognising penalty-sized numbers in the output.
The per-step objective is untouched. There the residuals really are a dimensionless fraction of typical motion and
0.1means what it says, so the RMS bar still decides; gated onscore_observed_only.Worth a reviewer's attention
reproduced_cascadeasks whether any candidate reproduced the cascade, not whether the RMS-best one did. Those differ, and "could this model ever produce this outcome" is a question about the whole grid.min_explainable_fitsgains a third return and the explainability cache a third element.Verification
177 tests pass, mypy clean over
code_sim_learning/, pylint 10.00/10. The new test fails under a revert of the structural switch — and exactly that one test fails, since it passes its ownSysIdConfigrather than touchingCFG, so interval scoring does not leak into the rest of the module.Not yet run end to end: the replay (
exp_domino_real_replay.yaml, added in #143) is the cheap way to see whether this turns the pinned 0.24 into an actual fit.