feat(qa): keep a failed multi-turn walk in the trace and in the denominator (refs #718)

[atomchung]

Smallest vertical slice of #718's promoted QA-repair front: make one consider route run observable enough that a failed multi-turn experience cannot disappear.

Head SHA: 41d1aab (branched from origin/main@a33f315). Three commits: the slice, then two rounds of fixes — see the independent-review disposition comment for what changed and why.

What changed

One route, driven twice, with the turns around it recorded. The walk invokes the real consider CLI on the scenario's declared motive, then again on the synthetic user's correction. No second state machine, no product renderer, no new route or persona. The engine's own challenge is what proves the correction landed: the second evaluation must carry the user's corrected words in challenge.quote_verbatim or the walk refuses.

An ordered visible-turn trace. qa/turn_trace.py records each surface with role, #718's message-type taxonomy, route state and timestamp, and chains their digests into ordered_surface_digest. The message type is derived from the surface, never declared by the fixture — a turn that narrates diagnosis is stored as process_error even though it arrived in the slot where a decision result was due.

Accounting that a failure can re-bucket but not erase. campaigns_started and route_runs_started reach disk before the first turn exists, and a started run reads harness_incomplete until a terminal verdict replaces it. A crash after the correction therefore leaves a partial trace, a turn count and a stop reason rather than no run. Exit codes separate the outcomes: 0 passed, 1 the product surfaces failed, 2 never settled.

The controlled regression. consider-ai-momentum-correction-mutated.json differs from the baseline in the third surface alone — a test asserts turns one and two are byte-identical — so what it catches is a delivery failure and nothing else. It is classified product_fail with all three turns retained.

Privacy split. Raw surface text goes to turn-trace.local.jsonl and machine detail to harness-diagnostics.local.json; the combined receipt carries counts, labels and digests only. A test asserts no raw surface reaches the public report, and that the visible route-failure surface carries none of the tool output behind it.

Evidence

python3 tests/test_synthetic_walk.py            25/25 ok
python3.11 tests/run_all.py                     PASS: all 55 suites passed
python3.12 tests/run_all.py                     PASS: all 55 suites passed

Mutation proof — 17 mutations across two batches, each restored by cp with __pycache__ cleared, all 17 caught. The first batch:

mutation	caught by
start_route stops assuming the run is incomplete	crash-retention test
a route crash is booked as a product failure	route-failure test
the crash path no longer flushes the partial trace	crash-retention test
every surface passes classification	denominator test
the narration marker list is emptied	classifier test
turns are sorted before chaining the digest	reorder test
tool output leaks into a visible surface	privacy test
the failing surface is still labelled a decision result	mutation test
the public index carries raw surface text	privacy test
a script may label its own message type	loader test
the correction need not appear in what the user said	loader test
a product failure exits like a pass	denominator test

Two corrections made during that proof, both recorded because they change how much the numbers are worth. The first readout of this dance was void: the probe grepped stdout case-insensitively for error, which matches a test name containing it, so all twelve REDs were unearned. Re-run on exit code alone. That run then left one survivor — mixing a position index into the digest chain is redundant, since a streaming hash already carries order — so the redundant field was removed rather than left untested, and the mutation replaced with one that sorts the turns.

Limits and rollback boundary

• The assistant surfaces come from a committed fictional script, not a host capture. production_answer_capture stays unavailable and ux_receipt_status stays incomplete_product_delivery, unchanged from before this PR.
• The classification is a delivery check — is this a decision result at all — not an answer-quality judgment. The eval: add first TradeEvaluation LLM judge slice #705 semantic judge is not called; semantic_judge stays skipped with a stated reason.
• The narration detector is English-only. qa/ is held to English by docs/language-policy.md, and that gate's allowlist is keyed on a line prefix, which for a list of quoted needles would be " — wide enough to retire the gate for the whole module. The walk pins --language en for the same reason; the lane is blind on the zh locales the product ships.
• owner_acceptance remains owner_unreviewed; nothing here maps a classification onto acceptance.
• Rollback: additive QA tooling only. qa/turn_trace.py and qa/scenarios/ are new; reverting this commit restores the previous single-invocation walk with no product-runtime change to undo. No engine, skill, schema or receipt-contract file is touched.

Not in scope

No dashboard, no parallel scheduler, no second persona or route, no #735 fan-out.

🤖 Generated with Claude Code

[atomchung]

Independent review — 8 findings, 5 real, all closed

An independent review (different model family, judged the diff before reading the PR body) ran against 61f34b8. Five findings were real defects with reachable scenarios; the fixes are in b273953 and 41d1aab. Head SHA is now 41d1aab.

#	Finding	Disposition
1	main()'s error path built a fresh ledger, so a run that died mid-trajectory printed route_runs_started: 0 on stdout	Fixed. run_walk hands the caller its last durable report through a sink; stdout and disk now agree. This was the slice's own failure mode on the surface most likely to be read.
2	A surface script declared its own terminal_options, so a fixture could invent a resolution word and manufacture a passing decision (terminal_options: ["banana"] + final text "FICTIONAL-A banana")	Fixed. The vocabulary now comes from schemas/trade-evaluation.schema.json; a script may declare a subset of the engine's own enum and nothing else.
3	Campaign/route start not durably recorded before a run directory exists	Accepted as a limitation. With finding 1 fixed, stdout reports campaigns_started: 1 honestly on that path; durable cross-run recording needs a campaign-level ledger, which is #735's aggregation, not this slice.
4	A failed write after settle() could leave a terminal verdict announced as recorded	Fixed. The sink is set only after both writes land, so the reported state is always a state that reached disk.
5	str(exc) carried raw engine stdout/stderr into the public report's error field	Fixed. Detail goes to stderr and the diagnostics artifact; the receipt keeps error_type only. This was a real privacy-boundary breach — tool internals in the part that may be posted.
6	The local trace was rewritten wholesale, so a death mid-write could truncate it	Fixed. Reports and traces write through a temp file and rename.
7	question and limitation are declared in the taxonomy but never derived	Accepted and documented. The taxonomy is the contract from comment 9, so the values stay; the docstring now names the cost — on this trajectory an assistant surface that asks a question is recorded as process_error, correctly a failure but under a label that says the wrong thing about why.
8	--plan's consider_invocations: 2 is a constant that would stay green with the second route call removed; the crash test bypassed main()	Fixed. A test now counts the real _consider calls and pins the plan constant to that count; another drives the CLI error path end to end.

Also fixed before the review landed, found while re-reading the exit-code contract: an exception outside the named list (JSONDecodeError from a malformed engine payload is the reachable one) reached the interpreter and exited 1 — the code reserved for "the product surfaces failed" — so a harness fault read as a product verdict. main() now catches every Exception and exits 2.

Evidence after the fixes

python3 tests/test_synthetic_walk.py       25/25 ok
python3.11 tests/run_all.py                PASS: all 55 suites passed
python3.12 tests/run_all.py                PASS: all 55 suites passed

Mutation dance, second batch: 5 more mutations for the new guards, 4 caught immediately and 1 survived — setting the report sink before the writes land, which no test could see without a write-failure scenario. Rather than call it covered, a test was added that fails the write of the settled report and asserts the announced state is the last durable one. That mutation is now caught. 17/17 across both batches, restore verified clean each time.

Two things worth stating plainly, since they change what the mutation numbers are worth: the first batch's readout was void (the probe grepped stdout case-insensitively for error, which matches a test name containing that word, so all twelve REDs were unearned — re-run on exit code alone), and one survivor in that batch turned out to be an equivalent mutation, so the redundant field it targeted was deleted rather than left untested.