diff --git a/AI_SETUP_LANES.md b/AI_SETUP_LANES.md index 2e1c506b..949c225f 100644 --- a/AI_SETUP_LANES.md +++ b/AI_SETUP_LANES.md @@ -11,12 +11,16 @@ This is **guidance, not a hard rule**. Maintainer override is always allowed. | **Advisor** | Fable 5 (via `advisorModel: "fable"`) | `high` (server-side) | | **Driver** | Sonnet 5 (`claude-sonnet-5`) | `medium` default, escalate `high` → `xhigh` for hard tasks | | **Reviewer** | Codex (GPT-5.6 Sol) xhigh | — | -| **Escalation** | Opus 4.8 xhigh or Fable 5 review | When stuck or high-stakes | +| **Escalation** | Opus 4.8 xhigh takes over as driver (or run a Fable 5 review pass) | When stuck (2 failed attempts / LOW confidence) or high-stakes | The new standard. Sonnet 5 beats Opus 4.6 on every coding benchmark (SWE-bench Verified 85.2% vs 80.8%, Terminal-Bench 80.4% vs 65.4%) and generally uses less Max quota — but the savings are not a fixed ratio: Sonnet 5's newer tokenizer produces ~30% more tokens for the same text than Opus 4.6's (per Anthropic's pricing docs), and community cost reports suggest the advantage narrows at `high`/`xhigh`. No controlled Sonnet-5-vs-Opus-4.6 quota measurement exists — check your own burn with `/usage`. Fable 5 advises at key decision points via native `advisorModel` (v2.1.170+). GPT-5.6 Sol xhigh reviews cross-family. Escalate to Opus 4.8 xhigh for the hardest debugging or architecture decisions — don't run Opus as the daily driver. **Effort escalation ladder:** Start at `medium` — CodeRabbit's testing found it captures most of Sonnet 5's upside at the lowest cost. Raise to `high` when medium struggles, `xhigh` for hard debugging, multi-file migrations, or long agent runs. `max` is rarely worth it — doubles cost for marginal gains per the same CodeRabbit testing. +**Two escalation axes — don't conflate them.** The effort ladder above raises reasoning *within* the Sonnet 5 driver. Model escalation is a different move: it *swaps the driver* — after 2 failed attempts, LOW confidence, or on a high-stakes change, Opus 4.8 xhigh takes over as driver (or run a Fable 5 review pass on the diff). Cranking Sonnet's effort past `xhigh` is not the escalation path. + +**Advisor failure has a fallback, not a shrug.** `advisor()` is a server-side tool and does fail during API incidents. When it errors, spawn a Fable subagent as the fallback reviewer — the same rule the `/sdlc` skill carries ("if down, spawn Fable subagent"). The advisor check is never skipped; only its transport changes. + **Requires:** Claude Code v2.1.197+ (Sonnet 5 alias resolution), Fable 5 access for advisor. ## Setup B — Opus 4.6 Stability (Legacy Flagship) @@ -142,10 +146,10 @@ If the advisor returns "Advisor unavailable," the server-side harness failed to **Step 1 — restart the session.** Exit and run `claude` (not `--resume`). A fresh process re-initializes the server handshake. This resolves most advisor failures. -**Step 2 — if the API incident persists:** +**Step 2 — if the API incident persists, swap the transport, not the check:** -- Continue with your driver model and no advisor — `/model sonnet` for Setup A, `/model claude-opus-4-6` for Setup B. Interactive — stays on your Max subscription. -- Or proceed without the advisor and let the Codex xhigh PR gate catch issues. +- Spawn a Fable subagent as the fallback reviewer — the `/sdlc` skill's standing rule ("if down, spawn Fable subagent"). Batch your plan or open questions into one subagent consult at each point where you'd have called `advisor()`. Runs interactively on your Max subscription like any other agent. +- Keep driving with your lane's model (`/model sonnet` for Setup A, `/model claude-opus-4-6` for Setup B). The subagent replaces the advisor's transport; the Codex xhigh PR gate remains the separate final backstop, not a substitute for the advisor check. **Last resort (scripted/CI only):** diff --git a/README.md b/README.md index 7e1727dd..8137c431 100644 --- a/README.md +++ b/README.md @@ -190,6 +190,14 @@ The wizard defines four AI coding setups in [`AI_SETUP_LANES.md`](AI_SETUP_LANES Setup D's whole point: **the discipline of knowing when NOT to use discipline.** When blast radius is low and you just need fast cheap hands, skip the SDLC overhead. +### Reading Setup A precisely + +Clarified 2026-07-13 after these exact points kept getting re-confused; each rule states its why: + +- **Effort escalation stays inside the driver.** Sonnet 5 starts at `medium`; `/effort high` when it struggles, `xhigh` for hard debugging or long agent runs. Why `medium` and not `high`: the old "start at `high`" guidance was removed in ROADMAP #440 — it had no measurement behind it, and CodeRabbit's testing (the source this repo already cites) found `medium` captures most of Sonnet 5's upside without paying for the top effort tiers, while community cost reports suggest the quota advantage erodes at `high`/`xhigh`. +- **Model escalation swaps the driver.** After 2 failed attempts, LOW confidence, or on high-stakes changes, Opus 4.8 xhigh takes over as driver (or run a Fable 5 review pass on the diff). Why a swap and not more effort: the lane's policy treats repeated failure as a sign the *approach* needs different eyes, not deeper reasoning on the same track — so once the effort ladder is exhausted, the next rung is a different model, not a bigger bill. +- **Advisor failure has a fallback, not a shrug.** Fable 5 advises via `advisorModel: "fable"`; when `advisor()` errors (it's a server-side tool — API incidents happen), spawn a Fable subagent as the fallback reviewer, exactly as the `/sdlc` skill prescribes. Why: the advisor's job is catching wrong approaches *before* they're built, so a transport failure changes how the advice is obtained — not whether the check happens. + **A note on `[1m]` and billing.** Sonnet 5 always runs at its native 1M context — no `[1m]` suffix needed, no separate billing tier. For Opus, the `[1m]` suffix is the 1M-context alias; as of [March 2026](https://claude.com/blog/1m-context-ga), 1M context is GA at standard pricing — **no long-context surcharge, no premium tier, no API-only restriction.** Interactive Claude Code sessions on Max / Team / Enterprise plans include 1M context automatically. (Pro users need "Enable usage credits" turned on once.) The [June 15, 2026 billing split](https://codersera.com/blog/anthropic-june-2026-billing-change-claude-code/) moved *headless* surfaces — `claude -p`, Agent SDK, GitHub Actions, third-party apps — off the Max subscription onto a separate metered credit pool. Interactive Claude Code in your terminal stays on Max. Full details in [`AI_SETUP_LANES.md` § How Billing Works](AI_SETUP_LANES.md#how-billing-works--1m-context-max-plan-and-the-june-15-split). ## How It Works diff --git a/ROADMAP.md b/ROADMAP.md index 9cc6bbcd..299b7196 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -26,7 +26,8 @@ Add entries when a candidate is mid-evidence (one signal exists but doesn't yet | Advisor auto-fallback in SDLC skill | **TRIGGER FIRED 2026-06-11.** When `advisor()` fails, the SDLC skill should automatically spawn a Fable subagent as fallback — proved in v1.82.0 triage (batch-reviewed 4 issues, caught #395 already fixed). Currently this pattern lives only in private memory; needs to be in `skills/sdlc/SKILL.md` so every project gets it. Also: auto-try advisor before every plan, not just when user asks. | Confidence ramp trial ships clean | 2026-07-11 | v1.82.0 triage session 2026-06-11 | | Detect process-rule memories in /feedback or /setup | Users saving memories like "always run tests" are patching /sdlc gaps with private memory. /feedback or /setup could detect `type: feedback` entries with process-rule patterns and suggest contributing back to /sdlc instead. | Maintainer runs memory audit and finds ≥3 process-rule memories | 2026-08-11 | v1.83.0 session 2026-06-12 | | Copilot Cowork SDLC port | GitHub Copilot desktop app launched in technical preview (June 2, 2026) with parallel multi-agent in git worktrees. Could port SDLC wizard as a Copilot extension. Different plugin format from Claude — would be a new sibling repo like `codex-sdlc-wizard`. Low priority: no demand signal, Copilot extension ecosystem is immature. | Second external user asks OR Copilot extension format stabilizes with hooks equivalent | 2026-08-20 | v1.84.0 triage session 2026-06-20 | -| Run /insights with Fable on this repo | Native CC `/insights` generates session usage report (friction patterns, satisfaction, outcome summaries). Run once with Fable model at max effort to get a high-quality analysis of how the wizard is actually being used. Review the HTML report for patterns that could improve /sdlc or hooks. | Next Fable session | 2026-07-22 | v1.83.0 session 2026-06-12, re-dated 2026-06-22 triage | +| ~~Run /insights with Fable on this repo~~ | **TRIGGER FIRED 2026-07-13.** Ran on Fable at max effort; global report generated + all 22 session facets analyzed per-repo. Findings: `/insights` is global-only (per-repo auditing stays in the filed GH issues — it can't do that job); the 30-day `cleanupPeriodDays` default had already deleted all history for ~15 project dirs, so **retention, not analysis, is the coverage bottleneck** (fix: raise `cleanupPeriodDays`); 8/11 surviving real sessions fully achieved, every failure environmental (Apple sync, npm perms, sandbox) not comprehension; the one recurring Claude-side friction family (overconfidence vs user-presented evidence, 3 incidents) already has 2 of 3 memory rules institutionalized. No /sdlc or hook changes warranted from this pass. | ~~Next Fable session~~ **FIRED** | — | v1.83.0 session 2026-06-12, re-dated 2026-06-22 triage | +| Rename research: "sdlc-wizard" → "SDLC harness" / "agentic harness"? | Maintainer signal 2026-07-13: "SDLC-wizard might not work [as a name] — maybe SDLC harness or agentic harness following SDLC — we can workshop it later." The product has arguably outgrown "wizard": it's an enforcement harness (blocking hooks, codex gate, cross-model review loops, drift tests), not a one-shot setup dialog. Tension to resolve: the xdlc ecosystem preamble explicitly says sdlc-wizard "earned the wizard form" — a rename must engage that history, not ignore it. Blast radius is large (repo name, npm package, sibling repos' inherited docs, README/ARCHITECTURE, xdlc references), so this needs a naming workshop + migration plan, not a drive-by edit. Workshop only — no rename before candidates are compared and consumer impact is mapped. | Maintainer schedules the naming workshop OR a second person independently reports name confusion OR next major-version planning starts | 2026-09-11 | This session 2026-07-13 (post-#440 wrap-up) | | Memory audit + repo efficiency pass with Fable | Run Memory Audit Protocol on private memory files — promote portable lessons, delete stale entries. Then: Fable batch-review the full repo (SKILL.md files, hooks, wizard doc) for efficiency, dead code, stale references, and consolidation opportunities. Goal: build a 95% confidence task list of cleanup items that can be worked in parallel. | Next Fable session | 2026-07-22 | v1.82.0 triage session 2026-06-11, re-dated 2026-06-22 triage | | ~~Sync AI Setup Lanes to Claude-family sibling wizards~~ | **DONE 2026-06-11.** gdlc v0.3.0, rdlc v0.7.0 shipped with AI Setup Lanes v2. Cowork plugin port in PR #410. Originally: `claude-gdlc-wizard` (v0.2.2) and `claude-rdlc-wizard` (v0.6.1) have stale 69-70 line AI_SETUP_LANES.md vs sdlc-wizard's 217 lines. Missing: 3-lane structure (Premium/Saver/Lite), advisor fallback escalation, Fable effort guidance, usage signals, autocompact cross-ref. Each needs a tailored port — gdlc is game-dev domain, rdlc is research domain. Codex/xdlc/ldlc out of scope (different ecosystem). When people run `/update` in those repos they should get the same Premium lane experience. | Next release of either sibling | 2026-07-11 | v1.82.0 triage session 2026-06-11 | diff --git a/tests/test-doc-consistency.sh b/tests/test-doc-consistency.sh index b5e87717..c958b366 100755 --- a/tests/test-doc-consistency.sh +++ b/tests/test-doc-consistency.sh @@ -1255,12 +1255,13 @@ test_ai_setup_lanes_reviewer_is_gpt56() { local bad="" # "5\.6" AND "Sol" (not just one or the other) so a Sol->Terra swap, or a # future GPT-5.7 Sol rename, both fail. - for n in 13 16 28 41 43 119 164 168 206 207 210; do + # (Line numbers re-pinned +4 after the 2026-07-13 Setup A clarity insertion.) + for n in 13 16 32 45 47 123 168 172 210 211 214; do bad="$bad$(_check_line_has_and_lacks "$F" "$n" "5\.6,Sol" "5\.5")" done - # L123 is the fallback-chain line: must name "5\.6" AND BOTH Sol (primary) + # L127 is the fallback-chain line: must name "5\.6" AND BOTH Sol (primary) # and Terra (fallback target) so a Terra->Luna swap also fails. - bad="$bad$(_check_line_has_and_lacks "$F" 123 "5\.6,Sol,Terra" "5\.5" "5\.4")" + bad="$bad$(_check_line_has_and_lacks "$F" 127 "5\.6,Sol,Terra" "5\.5" "5\.4")" if [ -z "$bad" ]; then pass "AI_SETUP_LANES.md: all reviewer-model lines reference GPT-5.6 Sol/Terra, none reference stale GPT-5.5/5.4" else @@ -1474,6 +1475,46 @@ test_sonnet5_default_effort_is_medium test_no_unsupported_sonnet5_sweet_spot test_opus46_max_sweet_spot_guard +# #440 follow-up (maintainer ask 2026-07-13): Setup A's two escalation axes and +# the advisor-failure fallback kept getting re-confused ("sonnet 5 xhigh? or +# high?" / "sometimes advisor fails then we need a sub agent remember"). +# Both README and AI_SETUP_LANES must state, explicitly: +# (a) model escalation SWAPS THE DRIVER (Opus 4.8 xhigh takes over) — it is +# not a higher rung on Sonnet's effort ladder; +# (b) when advisor() errors, the fallback is spawning a Fable subagent — the +# check is never skipped (same rule the /sdlc skill already carries). +test_setup_a_escalation_and_advisor_fallback_explicit() { + local bad="" + grep -q 'takes over as driver' "$REPO_ROOT/AI_SETUP_LANES.md" \ + || bad="$bad AI_SETUP_LANES.md:driver-swap" + grep -q 'spawn a Fable subagent' "$REPO_ROOT/AI_SETUP_LANES.md" \ + || bad="$bad AI_SETUP_LANES.md:advisor-fallback" + grep -q 'takes over as driver' "$REPO_ROOT/README.md" \ + || bad="$bad README.md:driver-swap" + grep -q 'spawn a Fable subagent' "$REPO_ROOT/README.md" \ + || bad="$bad README.md:advisor-fallback" + # Negative half (Codex round-1 P1-3): positive assertions alone false-green + # while the advisor-outage procedure still offers a "no advisor" path. The + # outage section must route to the subagent fallback, never to skipping. + local outage + outage="$(sed -n '/^## When the Advisor Is Unavailable/,/^## [^W]/p' "$REPO_ROOT/AI_SETUP_LANES.md")" + if [ -z "$outage" ]; then + bad="$bad AI_SETUP_LANES.md:outage-section-missing" + else + printf '%s' "$outage" | grep -qiE 'no advisor|without the advisor' \ + && bad="$bad AI_SETUP_LANES.md:outage-still-offers-skip-path" + printf '%s' "$outage" | grep -q 'Fable subagent' \ + || bad="$bad AI_SETUP_LANES.md:outage-missing-subagent-fallback" + fi + if [ -z "$bad" ]; then + pass "Setup A: driver-swap escalation + advisor subagent fallback explicit in README and AI_SETUP_LANES" + else + fail "Setup A clarity missing:$bad" + fi +} + +test_setup_a_escalation_and_advisor_fallback_explicit + # ──────────────────────────────────────────── # Summary # ────────────────────────────────────────────