fix(deep): inc-7 accuracy — infra-agnostic RCA with skill-driven health/failure gates - #261
Merged
Conversation
Final link in the chain: the orchestrator now forms the right Consul hypothesis (#256) with a verifiable prediction (#257), but the evidence gather queried the BARE consul_health_service_status metric — which returns one series per (node x status), mixed 0/1 — so the extractor produced zero usable observations and the keystone returned 'absent' ('couldn't verify'). Trace: impala runs kept proposing 'impala-statestore unhealthy' then gathering +0 observations. Fix: planPredictionQuery now appends a Consul-aware hint to the metric-threshold gather prompt — query it AGGREGATED for the service in the hypothesis: max by (service_name) (consul_health_service_status{service_name="X",status="passing"}) → one clean value (1 passing / 0 failing) the keystone can verify → confirm. Only triggers when the metric is consul_health_service_status (no false positives for ordinary metrics). tsc clean; 2 new tests + full suite green. Closes the form→predict→gather→verify chain for bare-metal Consul incidents.
Trace of a bd-management run showed the keystone CONFIRMING a true-but-wrong cause:
'bd-management is not deployed in the cluster (no k8s pod exists)'. It's true (a
Consul service has no k8s pod) but it is NOT the root cause — and the run had
already gathered consul_health evidence proving the service is bare-metal Consul.
The skill tells the model not to do this, but it does ~half the time; a prompt
can't guarantee it.
Deterministic guard in the conclude path: if the run gathered any
consul_health_service_status evidence AND the confirmed cause asserts a missing
k8s deployment ('not deployed', 'no k8s pod', 'deployment missing/absent'), reject
it and make the agent confirm via the Consul health signal instead. Genuine k8s
incidents (e.g. agw-admin-ui's deleted namespace) gather no consul_health
evidence, so the guard never fires for them — proven by a control test.
tsc clean; 2 new tests (guard fires for Consul / allows genuine k8s); suite green.
Validation surfaced a second wrong-layer failure: investigating the idle minimax-m25-metrics-proxy, the orchestrator ruled out every real hypothesis, then chased its OWN query errors — 'datasource with UID loki not found' (MI350 has no Loki datasource configured) appearing as 404s in logs — and confirmed 'Grafana datasource misconfigured' as the root cause. That's a query-layer artifact, not why the service is unhealthy. Guard (same pattern as the Consul category-error guard, same conclude path): reject a confirmed cause that blames the observability tooling (mentions 'datasource' or 'grafana') UNLESS the incident service is itself an observability component (grafana/loki/prometheus/...). Forces the agent back to the service's own signals; an idle service with no real cause then exhausts/pauses honestly instead of inventing a tooling cause. tsc clean; 2 new tests (guard fires for a service / allows a grafana incident); suite green.
Investigating minimax-m25-vllm-bench-4gpu (inv_01KTS9EK21G042S6YAS5MSCTE7), every
log query came back empty and the orchestrator kept reporting 'couldn't verify (no
evidence gathered)'. Root cause: gpt-oss generates an OR-chain of repeated line
filters to search multiple terms —
{container_name="svc"} |= "replicas" or {container_name="svc"} |= "scale" or {container_name="svc"} |= "deployment"
which is INVALID LogQL (`or` joins label-filter expressions, not `{sel} |=`
line-filter pipelines). Loki rejects it with HTTP 400 ('parse error ... unexpected
{'), so the gather returns ZERO log observations → no log evidence for any
log-based hypothesis (and contributes to MI350 investigations going inconclusive).
Fix: coerceLokiArgs (already on the Loki tool exec path, tool-utils.ts:345) now
detects the repeated-selector OR-chain and collapses it into the correct single
regex line filter — {sel} |~ "replicas|scale|deployment". Single filters and
already-regex queries pass through untouched; terms are deduped + regex-escaped.
tsc clean; 3 new tests; full suite green.
Symptom: log queries 400 → 'no evidence gathered'. Root cause: invalid OR-chain
LogQL. Fix: tool-utils.ts coerceLokiArgs. Regression: tool-utils.test.ts:84.
…-pause
Investigating minimax-m25-vllm-bench-4gpu (a k8s Deployment scaled to 0 replicas,
kube_deployment_spec_replicas=0), the orchestrator spent all ~30 moves chasing
Consul hypotheses ('is it a bare-metal Consul service / not registered in Consul /
health check unknown...'), never proposed the obvious 'deployment scaled to 0',
and operator-paused. Root cause: #256 injects the Consul runbook into EVERY
orchestrator run (always-on), which over-steers k8s services into Consul
tunnel-vision — a regression for the k8s case introduced by the Consul fixes.
Two parts:
- Skill: lead the 'When investigating' section with DETERMINE SERVICE TYPE FIRST —
query kube_deployment_status_replicas; if it returns data the service IS a k8s
Deployment (0 = scaled to 0 = a valid root cause, conclude it), and ONLY treat as
Consul when there's no kube_deployment metric. Stops the Consul tunnel-vision.
- Guard: the Consul category-error guard no longer fires when the run also saw
kube_deployment/kube_pod evidence — a real k8s Deployment's k8s-absence cause is
legitimate (only consul_health-AND-no-kube services are the bare-metal ones).
tsc clean; 1 new test (k8s service not blocked); suite green.
This was referenced Jun 11, 2026
The real flaw behind the k8s over-steer: #256 injected the bare-metal/Consul runbook into EVERY orchestrator run (getAllForScope always-on). So a Kubernetes service (e.g. minimax-m25-vllm-bench-4gpu, scaled to 0 replicas) got Consul context it should never see and the agent burned its whole budget on Consul hypotheses, never diagnosing the obvious k8s cause — exactly what the operator flagged. Fix: in handleOrchestratorInvestigate, drop infrastructure-type-specific runbooks (tagged consul/bare-metal) UNLESS the incident service is actually Consul-tracked — i.e. discovery recorded a consul_health_service_status metric for it. Signal is already in the registry: impala's metric is consul_health_service_status (Consul), minimax's is kube_deployment_*/vllm (k8s). No hardcoded service names (avoids the fazbd* product names in the live Consul list), auto-correct as discovery updates. A k8s investigation now has ZERO Consul in its context. Consul services still get the runbook. tsc clean; 1 updated + 1 new ws-handler test; suite green.
…skill The orchestrator and ws-handler had hardcoded Consul-specific logic (a conclude-time category-error guard keyed on consul_health_service_status, and a skill-injection scope that special-cased the Consul runbook). That coupled the engine to one infrastructure type. Replace it with a generic mechanism: a skill declares an optional `appliesToServiceMetric` substring in its frontmatter, and the engine injects that skill only when the incident service's discovered metric queries contain it. The Consul runbook now declares `appliesToServiceMetric: consul_health_service_status` — all infra-type knowledge lives in the skill, none in the engine. - orchestrator.ts: drop the Consul category-error guard (keep the generic observability-artifact guard) - ws-handler.ts: generic appliesToServiceMetric filter, no consul_health regex - hypothesis-requery.ts: generalize the Consul aggregation hint to any bare (un-aggregated) metric selector - store.ts: parse + round-trip the appliesToServiceMetric field A k8s service (e.g. a Deployment scaled to 0) now receives zero Consul context and diagnoses cleanly; Consul services still get the runbook via the declared metric match.
The keystone could not confirm an absence. When the brain proposed the correct cause for a workload scaled to zero replicas, the prediction's gather returned no data (pod-runtime metrics vanish at zero replicas), so evaluatePrediction scored it 'absent' → 'couldn't verify', the correct cause was discarded, and the run drifted to a wrong cause that happened to gather evidence (GPU OOM, cross-service blame) or wall-clocked. Across repeated runs on the same scaled-to-zero incident, only the run that happened to phrase a luckily-verifiable prediction succeeded. Two generic, infra-agnostic verification paths for an absence: - (B) state metric that survives at zero: the decide-move prompt steers the brain to predict over a STATE metric (e.g. kube_deployment_status_replicas < 1) that still reports 0, instead of a runtime metric that disappears. - (A) explicit absence prediction: extend the existing log-pattern present:false mechanism to infra-status. evaluatePrediction confirms an absence only when infra evidence was actually gathered (mirrors the log-pattern haveLogs guard — never confirm on a query that may not have run). schemas.ts threads present through the decide-move parse; hypothesis-requery asks the gather to report the resource's actual state including its absence.
The absence-verification fix made 'scaled to zero' confirmable — which is
correct for a genuine k8s workload but ALSO let a Consul service be falsely
confirmed as 'a Kubernetes Deployment scaled to zero' (5x re-run: impala,
a consul_health-tracked service, hit this 1/5). With the hardcoded Consul
guard removed, nothing enforced that a conclusion's infra framing matches
the service's actual type.
Add a generic conclude-time guard: derive the service's infrastructure type
from its discovered identity metric FAMILY (kube_* → k8s, consul_* → Consul)
and reject a confirm whose framing contradicts it — a k8s/deployment/replica
cause for a Consul service, or a Consul cause for a kube_deployment workload.
Driven by the metric family, not a per-service or Consul-only list; silent
when the identity is an unrecognised family (up{}, etc.).
Threads the incident service's identity metrics (ws-handler → agents.orchestrate
→ runAutonomousOrchestrator → OrchestratorDeps). Also catches the reverse
hallucination seen in the re-run (a k8s service confirmed with a 'consul
registration' tail).
…blind) The orchestrator is LLM-driven and non-deterministic — the same incident can confirm the right cause, wall-clock, or confirm a wrong one across runs. Until now 'quality is bad' was a vibe from slow manual batches; there was no way to tell whether a change helped. This harness turns a batch of orchestrator runs into objective rates: - src/eval/deep-eval.ts — pure scorer + CLI. Scores each run against a label: correct / confident-wrong / category-error / honest-inconclusive. Headline metrics: correctRate and confidentWrongRate (the 'do no harm' bar, target 0). --save / --compare / --max-confident-wrong / --max-category-error / --min-correct. - src/eval/deep-investigation-run.mjs — live batch runner (baseline → orchestrator, auto-continue at operator-pauses), writes a results JSON the scorer reads. Makes the previously-manual batch reproducible. - fixtures/deep-investigation-labels.json — labeled incidents (infra type, expected cause keywords, wrong-cause patterns, acceptable inconclusive outcomes). - baselines/deep-2026-06-11.json — first scorecard after Fix 1: 40% correct, 20% confident-wrong (the GPU fabrication), 0 category-error. 10 scorer unit tests. Establishes the measurement loop to gate the un-gate on 0% category-error / 0% fabrication / correct-on-clear-cases.
Two measured-against-baseline accuracy fixes, both driven by the same incidentServiceMetrics identity signal Fix 1 introduced. Fix 2 — identity-steering (Consul conclusiveness). The brain often never proposed the hypothesis that works: for a Consul service it wandered into k8s framings or dependency-chasing instead of testing consul_health, and wall-clocked (impala 2/3). When it DID propose 'Consul health failing' it verified fine (+4 obs). serviceIdentityHint derives a one-line steer from the service's identity metric family and prepends it to the decide-move prompt: Consul service → test consul_health_service_status < 1 first; k8s workload → establish replica state first. Positive mirror of the Fix 1 guard. Fix 3 — grounding gate (kill the fabrication). The 20% confident-wrong in the baseline was minimax confirming 'GPU resource exhaustion' for a deployment scaled to zero — there are no pods at 0 replicas, so no GPU/OOM/readiness failure is possible (cluster-wide GPU usage misattributed). Conclude-time guard: if gathered evidence shows zero running pods, reject a confirm that blames a pod-runtime failure (GPU/CUDA/OOM/readiness/crash-loop/CPU). Only fires when an observation actually establishes zero replicas, so it's grounded. +6 unit tests; 2731 green. Re-score against baselines/deep-2026-06-11.json next.
…op stale baselines The MI350 stack was removed, so the minimax (k8s scaled-to-zero) label is retired. Re-key the eval labels to services present on the current bigdata cluster stacks: Consul services (impala, kudu, hdfs-namenode, bd-management-task-management) and k8s workloads (ingestion-server, bd-management-task). The k8s 'bd-management-task' vs Consul 'bd-management-task-management' pair is a direct category-error probe. Delete the MI350-era baselines (deep-2026-06-11.json, deep-latest.json) so the next run starts fresh; --save regenerates the baseline. CLAUDE.md updated to the --save / CI-gate command pair.
…fident-wrong) 8-incident run across the two live stacks. Corrected the labels: a Consul service can be Consul-healthy while failing on its own daemon or a real dependency, so a grounded on-type cause is CORRECT — only a cross-type category error or a fabrication is wrong. Added timeout-orch as an honest non-confirm. Result (deep-2026-06-15): 0% category-error, 0% confident-wrong, 0 fabrications. 3 grounded confirms (impala daemon, ingestion-server CPU, Cortex ingester disk), 4 honest declines, 1 excluded (network baseline-fail). Both Consul confirms tried consul_health FIRST then ruled it out when healthy (Fix 2 working). Meets the un-gate safety bar on real cluster traffic.
The single 'Consul Bare-Metal Service Discovery' skill was doing double duty (scope: [discovery, investigation]) — its title says discovery but it also carried the RCA/investigation runbook. Split into two single-purpose skills so Consul knowledge stays in skills (never the engine) and each is scoped right: - consul-bare-metal-discovery (scope: discovery) — finding Consul services + writing registry entries, for the discovery agent. No appliesToServiceMetric. - consul-bare-metal-investigation (scope: investigation, appliesToServiceMetric: consul_health_service_status) — RCA runbook, injected into the orchestrator ONLY for Consul-tracked services when enabled. Carries the verifiable confirm prediction (aggregated passing form, so the gather reduces to a value instead of 'couldn't verify') AND an explicit 'healthy != a root cause' rule: never confirm 'Consul failing' when the passing row reads 1 — decline instead. This is the skill-side home for the consul-health verification (no engine hardcoding) and addresses the ground-truth miss + the over-confirm on healthy Consul services. 2731 tests green.
…split) P1b: the k8s-side 'healthy != a root cause' runbook, parallel to the Consul investigation skill. investigation-scoped, appliesToServiceMetric: kube_ — so it's injected into the orchestrator only for k8s-tracked services. Carries the rule that fixes the ingestion-server false-confirm: never confirm CPU-throttling / OOM / readiness / crash causes unless the actual throttle/restart/OOM metric was gathered and crossed a threshold; if replicas available == spec and nothing abnormal was seen, decline (inconclusive) rather than manufacture a cause. Keeps infra knowledge in skills, engine generic. 2731 tests green.
…mpt assembly No prior visibility into whether investigation skills actually reach the decide-move prompt. Add LOG_LEVEL=debug logs at both points: ws-handler logs the matched skills (+ appliesToServiceMetric) and skillContext size for the incident service; orchestrator-llm logs whether skillContext/identityHint are present and the injected skill titles. Used to prove the Consul investigation skill IS injected for impala yet the LLM over-confirms anyway — i.e. skill prompts are followed unreliably, so a deterministic engine gate is needed.
…skills The decisive finding (debug-log proof): the investigation skill IS injected into the decide-move prompt, but the LLM routes around its 'healthy != cause' rule and over-confirms on healthy services. So prompt guidance alone can't hold — a deterministic engine gate is required — AND the engine still hardcoded Consul/k8s literals (one always-on in the SYSTEM_PROMPT, distracting non-Consul stacks). Fix both: move ALL infra-type knowledge into skill-declared metadata and have the engine read it generically. - store: skills declare healthySignal / identityHint / incompatibleClaims ($service substituted), parsed + round-tripped like appliesToServiceMetric. - consul + k8s investigation skills declare those fields. - orchestrator-llm: drop serviceIdentityHint (the consul/k8s detector); take identityHint from the matched skill; generalize the SYSTEM_PROMPT to 'don't assume a platform; use the injected skills' (removes the always-on Consul/k8s/vllm literals → no distraction on a non-Consul stack). - orchestrator: service-type guard now rejects on the matched skill's incompatibleClaims regex (no consul/kube literals); NEW health-gate evaluates the matched skill's healthySignal before any confirm — healthy => inconclusive (the deterministic backstop the proof demands). - agents.ts derives identityHint/incompatibleClaims/checkHealthy from the matched skills (+ providers for the healthySignal query); prometheus-query gains queryInstantValue. Result: orchestrator engine carries zero Consul literals (only comments + the eval term-list). A stack with no Consul skill enabled sees zero Consul text. 2732 tests green. (Residual: the k8s grounding-gate still has kube_ literals — follow-up.)
… filter Root cause of the ingestion-server over-confirm that survived the refactor: getAllForScope sliced to maxPerQuery (default 3) by TITLE before the ws-handler's appliesToServiceMetric filter ran. With 4 investigation skills (Consul…, Investigate High-CPU, Investigate Kafka, Kubernetes Workload…), 'Kubernetes Workload Investigation' sorts last and was dropped by the cap — so the k8s skill (and its healthySignal) never reached a k8s service, the health-gate never ran, and ingestion kept over-confirming. Fix: getAllForScope returns ALL scoped skills uncapped; ws-handler relevance- filters by appliesToServiceMetric, orders service-targeted skills FIRST, then caps to maxPerQuery — so a matched infra skill can never be dropped. Found via the skill-injection debug log. 2732 tests green.
… real-incident catch) The remaining gap after the infra-agnostic refactor: bd-management (consul CRITICAL) confirmed correctly one run and wall-clocked the next — LLM variance meant a genuinely-broken service wasn't reliably caught. Fix it deterministically + skill-driven. Skills declare a failureSignal (PromQL >=1 when the service is definitely failing on its primary signal — consul status=critical, replicas unavailable) and a failureCause (the grounded cause text). The engine evaluates failureSignal at every give-up point (wall-clock / exhausted / tool-cap / budget / inconclusive / operator-pause) via an async giveUp() wrapper; if it fires, the run confirms the skill's failureCause instead of missing the incident. Confirms and aborts are untouched; an explicit agent confirm is never overridden. No infra literals in the engine — failureSignal + failureCause live in the skills. +3 unit tests; 2735 green. Verify live next.
F1 — a generic SYSTEM_PROMPT rule: a rate/ratio/percentage/throughput cause
('0% success rate', 'high error rate', 'queries returning zero results') is only
valid with evidence of NON-ZERO traffic; zero requests = idle, not broken. No
infra literals. Complements the health-gate (which already blocks confirms on a
healthy service) for the general idle-misread.
F2 — ground-truth-anchored scoring, the metric that actually tracks quality
(keyword scoring gave a false PASS — it can't tell a grounded-sounding
false-confirm from a real cause). deep-eval.ts gains scoreRunsGroundTruth +
--ground-truth: score each run against the service's REAL health —
confirm-on-healthy = false-confirm, decline-on-broken = missed-incident (both
target 0), with --max-false-confirm / --max-missed gates. deep-investigation-
groundtruth.mts fetches the truth via grafana-mcp using the skills' own
healthySignal/failureSignal (consistent with the engine). +6 scorer tests,
including an aggregate that reproduces the real 8-run finding (2 false-confirms
+ 1 missed) — i.e. the harness would now catch what the keyword scorer missed.
WZ
changed the title
This was referenced Jun 18, 2026
WZ
added a commit
that referenced
this pull request
Jun 18, 2026
…absence-confirm (#263) * fix(deep): #261 review follow-ups — re-cap discovery skills, tighten absence-confirm Two P3 items deferred from the #261 review (now on merged main): - resolveDiscoverySkills: re-cap to maxPerQuery. getAllForScope no longer caps itself (the investigation path filters-then-caps), so the discovery path was silently de-capped and would inject every enabled discovery skill into the prompt. 0 impact today (1 discovery skill) but latent token bloat. - evaluatePrediction infra-status present:false: only confirm an absence when the gather actually observed the named resource. Previously any infra observation (even one about a different resource) let a 'resource absent' prediction read 'satisfied' — so a query that didn't cover the resource could falsely confirm 'scaled to zero'. Now: resource never observed → unknown, not confirmed. (#11 grounding-gate value===0 investigated and confirmed a non-issue: observation-normalize sets value via parseMetricValue(...) ?? ... so it is undefined when absent, never defaulted to 0; value===0 excludes undefined.) * fix(config): bound maxPerQuery/maxCharsPerSkill to non-negative ints Review follow-up: the discovery-skill cap (and the existing search/investigation caps) slice on maxPerQuery; a negative value would slice(0,-1) and silently drop the last skill, a float would truncate oddly. Constrain at the schema.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hardens the autonomous deep-investigation orchestrator (gated behind
agent.autonomousInvestigationEnabled/agent.challengeEnabled) for inc-7 accuracy. Guiding principle: the engine carries no infrastructure literals — all platform knowledge (Consul, Kubernetes, …) lives in team-knowledge skills, applied only when a skill is enabled and matches the incident service. Supersedes the three original inc-7 fixes #258 / #259 / #260 (now closed) and extends well past them.Engine is now infra-agnostic
orchestrator.tsinto skill frontmatter. Skills declareappliesToServiceMetric,healthySignal,failureSignal,failureCause,identityHint,incompatibleClaims; the engine reads them generically (zero infra literals in active code).maxPerQuerycapped the scoped skill set by title sort before the relevance filter, dropping a title-last targeted skill. Now: filter, then cap with service-targeted skills first.Deterministic confirm/decline gates (skill-driven)
healthySignalreads healthy, block the confirm and conclude inconclusive (a healthy service has no incident to explain). Evaluated at the incident window, not wall-clock.failureSignalfires, confirm the groundedfailureCauseinstead of missing a genuinely-broken service. Requires the service not read healthy simultaneously.incompatibleClaims(a cause that contradicts the service's platform).Evidence-gather repairs (the original #258–#260)
{sel} |= "a" or {sel} |= "b"(Loki HTTP 400 → no evidence gathered);coerceLokiArgscollapses it to{sel} |~ "a|b", only when every stream selector is identical.Ground-truth-anchored eval (F2)
New harness scores runs against each service's real health (confirm-on-healthy = false-confirm, decline-on-broken = missed-incident), captured per-run so a flapping service is judged at its run time. Keyword scoring previously gave a false PASS.
Review hardening
Windowed health/failure gates (incident time, not now), failure-floor healthy cross-check, gate-query timeouts so a hung metrics MCP can't defeat the wall-clock guard,
$servicesanitized before PromQL substitution, removed dead plumbing, and extracted pure helpers (deriveSkillGuards,selectInvestigationSkills) with unit tests.Status
tsc clean · full suite green (2763 tests) · feature still gated off. Final live ground-truth re-run pending (LLM gateway availability) before the un-gate flip.