Skip to content

feat(qa): synthetic consider user walkthrough - #723

Merged
atomchung merged 3 commits into
mainfrom
codex/issue-718-synthetic-user-walk
Aug 1, 2026
Merged

feat(qa): synthetic consider user walkthrough#723
atomchung merged 3 commits into
mainfrom
codex/issue-718-synthetic-user-walk

Conversation

@atomchung

@atomchung atomchung commented Aug 1, 2026

Copy link
Copy Markdown
Owner
  1. LLM role = synthetic user, never final-answer generator — qa/synthetic_user.py and tests/test_synthetic_walk.py constrain it to the current allowed action.
  2. Execute the real consider-ai-momentum walkthrough in isolated FOMO QA state — qa/run_synthetic_walk.py invokes review.py consider with isolated HOME and TRADE_COACH_HOME.
  3. Capture the exact product-visible answer byte-for-byte — skills/fomo-kernel/engine/consider_surface.py, review.py --product-surface, and the capture equality assertion.
  4. Build the deterministic schema-v3 artifact and call the existing judge — qa/candidate_builder.py, evals/judge_trade_answers.py --answer-file --source-fixture, and the existing eval: add first TradeEvaluation LLM judge slice #705 candidate validators.
  5. Stop after one scenario, one persona, and the consider route — tests/test_synthetic_walk.py pins consider-ai-momentum / ai-momentum-investor / consider.

Scope

  • One command defaults to a real agy synthetic user, runs the actual route, captures its final answer, emits the candidate artifact, calls the existing judge, and writes a combined receipt with workflow, deterministic, semantic_judge, and owner_acceptance=owner_unreviewed.
  • A live agy walk completed the real context queue, final product capture, and UX receipt using only isolated synthetic data. Its receipt has workflow=pass, deterministic=pass, and semantic_judge=skipped because that verification intentionally used --no-semantic-judge.
  • The offline test lane uses --user-backend stub --no-semantic-judge; the judge plan has been verified against the captured artifact. The runner records semantic_judge=unavailable if the existing judge backend cannot return, rather than dropping the combined receipt.
  • No second scenario, dashboard, benchmark, persona library, judge rubric, eval framework, README, or documentation expansion.

Verification

  • PYTHONPYCACHEPREFIX=/tmp/fomo-pycache python3 tests/run_all.py
  • python3 qa/run_synthetic_walk.py consider-ai-momentum --user-backend stub --no-semantic-judge --output-dir /tmp/fomo-synthetic-issue718
  • python3 evals/judge_trade_answers.py --answer-file /tmp/fomo-synthetic-issue718/candidate.json --source-fixture /tmp/fomo-synthetic-issue718/source-fixture.json --plan
  • PYTHONPYCACHEPREFIX=/tmp/fomo-pycache python3 qa/run_synthetic_walk.py consider-ai-momentum --user-backend agy --no-semantic-judge --output-dir /tmp/fomo-synthetic-live-718-final

Refs #718

@atomchung
atomchung force-pushed the codex/issue-718-synthetic-user-walk branch from 15da342 to a46f65e Compare August 1, 2026 13:55
@atomchung
atomchung marked this pull request as ready for review August 1, 2026 14:04
@atomchung
atomchung merged commit 3eb12f7 into main Aug 1, 2026
3 checks passed
@atomchung
atomchung deleted the codex/issue-718-synthetic-user-walk branch August 1, 2026 14:04

Copy link
Copy Markdown
Owner Author

Post-merge review finding — resolve before freezing #713

PR #723's isolated QA orchestration is useful and its Python 3.11/3.12 CI is green, but the merged cut also introduced a generic runtime/product surface that does not satisfy the contract it claims to capture.

P1 — --product-surface is scenario-specific copy exposed as a generic consider answer

skills/fomo-kernel/engine/consider_surface.py::render() unconditionally opens the case with:

  • “If the demand observation proves durable…”
  • “…changing the cap can be a deliberate choice.”

Those statements are not derived from the supplied premise, side, context or challenge. The public CLI flag is generic (review.py consider --product-surface) and accepts sell decisions, non-demand reasons, and cases with no cap-shaped tension. The focused production test covers only the one AI-momentum buy fixture, so this remains green while another valid consider scene receives a fabricated decision frame.

The same renderer is English-only and emits raw internal topic labels (The frozen {topic} fact...), so it is not the existing localized/output-voice product surface either.

Contract consequence

Issue #718 says this lane is additive maintainer tooling, not a runtime feature, and that product runtime behavior stays unchanged. It also requires capturing the exact existing product-visible answer byte-for-byte. The current implementation instead creates a new deterministic answer renderer inside the engine and then captures that new surface. Therefore either:

  1. this is a QA-only fixture renderer, in which case it must not be exposed or described as the generic product answer and cannot prove exact production-answer capture; or
  2. this is a new product answer path, in which case it needs a separate product owner, generic scenario/locale/output-voice coverage, and acceptance evidence beyond [qa] Add an LLM-driven synthetic FOMO QA walkthrough #718.

Smallest safe correction

Do not widen this into a response framework. Before #713 freezes a candidate:

  • remove the generic product claim/runtime exposure or constrain the adapter so it cannot render outside the declared synthetic scenario;
  • ensure the synthetic walk captures the actual product answer path rather than a scenario-specific substitute;
  • add a regression using at least one non-demand/sell consider scene that fails on the current hard-coded text;
  • preserve the isolated synthetic-user, exact-byte receipt, schema-v3 builder and existing judge work that is genuinely QA-only.

Until this boundary is resolved, main@3eb12f7 should not be named the #713 reliability candidate. Green CI proves the implemented behavior is stable, not that this is the correct product surface.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant