You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
LLM role = synthetic user, never final-answer generator — qa/synthetic_user.py and tests/test_synthetic_walk.py constrain it to the current allowed action.
Execute the real consider-ai-momentum walkthrough in isolated FOMO QA state — qa/run_synthetic_walk.py invokes review.py consider with isolated HOME and TRADE_COACH_HOME.
Capture the exact product-visible answer byte-for-byte — skills/fomo-kernel/engine/consider_surface.py, review.py --product-surface, and the capture equality assertion.
Build the deterministic schema-v3 artifact and call the existing judge — qa/candidate_builder.py, evals/judge_trade_answers.py --answer-file --source-fixture, and the existing eval: add first TradeEvaluation LLM judge slice #705 candidate validators.
Stop after one scenario, one persona, and the consider route — tests/test_synthetic_walk.py pins consider-ai-momentum / ai-momentum-investor / consider.
Scope
One command defaults to a real agy synthetic user, runs the actual route, captures its final answer, emits the candidate artifact, calls the existing judge, and writes a combined receipt with workflow, deterministic, semantic_judge, and owner_acceptance=owner_unreviewed.
A live agy walk completed the real context queue, final product capture, and UX receipt using only isolated synthetic data. Its receipt has workflow=pass, deterministic=pass, and semantic_judge=skipped because that verification intentionally used --no-semantic-judge.
The offline test lane uses --user-backend stub --no-semantic-judge; the judge plan has been verified against the captured artifact. The runner records semantic_judge=unavailable if the existing judge backend cannot return, rather than dropping the combined receipt.
No second scenario, dashboard, benchmark, persona library, judge rubric, eval framework, README, or documentation expansion.
Post-merge review finding — resolve before freezing #713
PR #723's isolated QA orchestration is useful and its Python 3.11/3.12 CI is green, but the merged cut also introduced a generic runtime/product surface that does not satisfy the contract it claims to capture.
P1 — --product-surface is scenario-specific copy exposed as a generic consider answer
skills/fomo-kernel/engine/consider_surface.py::render() unconditionally opens the case with:
“If the demand observation proves durable…”
“…changing the cap can be a deliberate choice.”
Those statements are not derived from the supplied premise, side, context or challenge. The public CLI flag is generic (review.py consider --product-surface) and accepts sell decisions, non-demand reasons, and cases with no cap-shaped tension. The focused production test covers only the one AI-momentum buy fixture, so this remains green while another valid consider scene receives a fabricated decision frame.
The same renderer is English-only and emits raw internal topic labels (The frozen {topic} fact...), so it is not the existing localized/output-voice product surface either.
Contract consequence
Issue #718 says this lane is additive maintainer tooling, not a runtime feature, and that product runtime behavior stays unchanged. It also requires capturing the exact existing product-visible answer byte-for-byte. The current implementation instead creates a new deterministic answer renderer inside the engine and then captures that new surface. Therefore either:
this is a QA-only fixture renderer, in which case it must not be exposed or described as the generic product answer and cannot prove exact production-answer capture; or
Do not widen this into a response framework. Before #713 freezes a candidate:
remove the generic product claim/runtime exposure or constrain the adapter so it cannot render outside the declared synthetic scenario;
ensure the synthetic walk captures the actual product answer path rather than a scenario-specific substitute;
add a regression using at least one non-demand/sell consider scene that fails on the current hard-coded text;
preserve the isolated synthetic-user, exact-byte receipt, schema-v3 builder and existing judge work that is genuinely QA-only.
Until this boundary is resolved, main@3eb12f7 should not be named the #713 reliability candidate. Green CI proves the implemented behavior is stable, not that this is the correct product surface.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
qa/synthetic_user.pyandtests/test_synthetic_walk.pyconstrain it to the current allowed action.consider-ai-momentumwalkthrough in isolated FOMO QA state —qa/run_synthetic_walk.pyinvokesreview.py considerwith isolatedHOMEandTRADE_COACH_HOME.skills/fomo-kernel/engine/consider_surface.py,review.py --product-surface, and the capture equality assertion.qa/candidate_builder.py,evals/judge_trade_answers.py --answer-file --source-fixture, and the existing eval: add first TradeEvaluation LLM judge slice #705 candidate validators.considerroute —tests/test_synthetic_walk.pypinsconsider-ai-momentum/ai-momentum-investor/consider.Scope
agysynthetic user, runs the actual route, captures its final answer, emits the candidate artifact, calls the existing judge, and writes a combined receipt withworkflow,deterministic,semantic_judge, andowner_acceptance=owner_unreviewed.agywalk completed the real context queue, final product capture, and UX receipt using only isolated synthetic data. Its receipt hasworkflow=pass,deterministic=pass, andsemantic_judge=skippedbecause that verification intentionally used--no-semantic-judge.--user-backend stub --no-semantic-judge; the judge plan has been verified against the captured artifact. The runner recordssemantic_judge=unavailableif the existing judge backend cannot return, rather than dropping the combined receipt.Verification
PYTHONPYCACHEPREFIX=/tmp/fomo-pycache python3 tests/run_all.pypython3 qa/run_synthetic_walk.py consider-ai-momentum --user-backend stub --no-semantic-judge --output-dir /tmp/fomo-synthetic-issue718python3 evals/judge_trade_answers.py --answer-file /tmp/fomo-synthetic-issue718/candidate.json --source-fixture /tmp/fomo-synthetic-issue718/source-fixture.json --planPYTHONPYCACHEPREFIX=/tmp/fomo-pycache python3 qa/run_synthetic_walk.py consider-ai-momentum --user-backend agy --no-semantic-judge --output-dir /tmp/fomo-synthetic-live-718-finalRefs #718