fix(deep): inc-7 accuracy — infra-agnostic RCA with skill-driven health/failure gates

CLAUDE.md

@@ -28,6 +28,14 @@ npm run test:discover-eval            # Score discovery output quality (gates at
 npx tsx src/eval/rca-eval.ts          # Score RCA report quality
 npx tsx src/eval/rca-eval.ts --save   # Score + save baseline
 npx tsx src/eval/rca-eval.ts --compare src/eval/baselines/2026-03-22.json  # Compare to baseline
+# Deep-investigation (autonomous orchestrator) quality: run a batch live, then score it
+node src/eval/deep-investigation-run.mjs <incidents.json> /tmp/runs.json     # live batch (server on :3000)
+npx tsx src/eval/deep-eval.ts --results /tmp/runs.json                        # score: correct / confident-wrong / category-error rates
+npx tsx src/eval/deep-eval.ts --results /tmp/runs.json --save                # writes baselines/deep-latest.json
+npx tsx src/eval/deep-eval.ts --results /tmp/runs.json --max-confident-wrong 0 --max-category-error 0   # CI gate (keyword)
+# Ground-truth-anchored (the metric that actually tracks quality — confirm-on-healthy = false-confirm):
+npx tsx src/eval/deep-investigation-groundtruth.mts <stackDir> > /tmp/gt.json   # fetch real health via grafana-mcp
+npx tsx src/eval/deep-eval.ts --results /tmp/runs.json --ground-truth /tmp/gt.json --max-false-confirm 0 --max-missed 0
 npx vitest run           # Run all tests
 npx vitest run src/path  # Run a single test file
 npx tsc --noEmit         # Type check
@@ -71,6 +79,7 @@ npx tsc --noEmit         # Type check
 | Shared PromQL parser | `src/lib/prom-metric.ts` — `extractMetricExpression()`, imported by both server metric extraction and the web MetricsPanel empty-card titles |
 | Investigation export helpers | `src/web/lib/exportInvestigation.ts` — `downloadPng` (html-to-image + font preload), `downloadMarkdown`, `copyMarkdown` |
 | RCA eval harness | `src/eval/rca-eval.ts` — scores RCA reports on 5 quality dimensions, baselines in `src/eval/baselines/` |
+| Deep-investigation eval | `src/eval/deep-eval.ts` (scorer: correct / confident-wrong / category-error rates) + `src/eval/deep-investigation-run.mjs` (live batch runner). Labels: `src/eval/fixtures/deep-investigation-labels.json`. The orchestrator is non-deterministic — this turns a manual batch into objective rates |
 | LLM quirk workarounds | `src/agents/shared/prepare-step.ts` (`prepareStep` hook) |
 | LLM retry & graceful failure | `src/agents/shared/llm-retry.ts` (`withLlmRetry`, `safeAgentRetryConfig`), `src/agents/shared/llm-errors.ts` (`LlmUnavailableError`, `isLlmUnavailable`). Tool-using agent paths only retry when `readOnlyTools: true` to avoid replaying write tool calls |
 | Shared types | `src/types/` — RCA report, agent interfaces, LLM types, WebSocket protocol |

skills/consul-bare-metal-discovery.md

@@ -10,24 +10,9 @@ tags:
   - bigdata
 scope:
   - discovery
-  - investigation
 ---
 ## When to use
-This stack runs services on bare-metal hosts (not K8s). They are registered in Consul and export health status via the `consul_health_service_status` metric.
-
-## When investigating a root cause (read this first)
-If a service appears down but has **no Kubernetes Deployment / Pod**, that is EXPECTED here — it is a bare-metal Consul service, not a k8s workload. Do NOT report "deployment missing" or "not deployed in the cluster" as the root cause. Its health signal is the Consul metric:
-```
-max by (service_name) (consul_health_service_status{service_name="<name>",status="passing"})
-```
-A value of `0` (or no row) means the bare-metal service is failing its Consul health check — that is the real signal to investigate (check the host process, its logs via the bare-metal logLabels, and any upstream it depends on). Only conclude a k8s cause for services that actually have k8s objects.
-
-### To CONFIRM it (so the test actually verifies, not "absent")
-When you hypothesize that a bare-metal service is unhealthy, attach this EXACT checkable prediction — the keystone matches the metric name literally, so use it verbatim:
-```json
-{"kind":"metric-threshold","metric":"consul_health_service_status","op":"<","value":1}
-```
-Then `query` it: the evidence gather runs that metric for the service and reports its value. If the passing-status value is `< 1` (i.e. 0), the `test` move returns **satisfied** → you can `conclude`. A vague hypothesis with no `consul_health_service_status` metric-threshold prediction will always come back "couldn't verify" (absent), so the run stalls — always make the prediction this exact metric.
+This stack runs services on bare-metal hosts (not K8s). They are registered in Consul and export health status via the `consul_health_service_status` metric. This runbook is for the **discovery agent** — finding those services and writing their registry entries. (Investigating a Consul service incident is a separate skill: "Consul Bare-Metal Service Investigation".)
 
 ## Discovery strategy
 

skills/consul-bare-metal-investigation.md

@@ -0,0 +1,67 @@
+---
+title: Consul Bare-Metal Service Investigation
+services: []
+alerts: []
+appliesToServiceMetric: consul_health_service_status
+healthySignal: 'max by (service_name) (consul_health_service_status{service_name="$service",status="passing"})'
+failureSignal: 'max by (service_name) (consul_health_service_status{service_name="$service",status="critical"})'
+failureCause: '$service is failing its Consul health check (status=critical) — the bare-metal host process is down or unhealthy.'
+identityHint: '$service is registered in Consul (identity metric consul_health_service_status) — it has NO Kubernetes Deployment by design. Investigate its Consul health (status="critical") and host process FIRST; do not form k8s hypotheses for it.'
+incompatibleClaims: 'kubernetes|k8s deployment|scaled to zero|scaled to 0|zero replicas|0 replicas|namespace'
+tags:
+  - investigation
+  - rca
+  - consul
+  - bare-metal
+  - host-process
+scope:
+  - investigation
+---
+## When to use
+This runbook applies ONLY to services registered in Consul on bare-metal hosts —
+identified by the `consul_health_service_status` metric. It is injected into an
+investigation only when the incident service is Consul-tracked and this skill is
+enabled. If the incident service is a Kubernetes workload (it has a
+`kube_deployment_*` metric), this runbook does not apply — investigate its
+deployment/pod state instead.
+
+## The health signal
+A bare-metal Consul service has **no Kubernetes Deployment/Pod by design** — do NOT
+report "deployment missing / not deployed in the cluster". Its health is the Consul
+check, which emits one row per (node × status). Read it aggregated:
+```
+max by (service_name) (consul_health_service_status{service_name="<name>",status="passing"})
+```
+- value `1` → the service is **passing** its Consul health check (healthy on this axis).
+- value `0` (or no passing row) → it is **failing** — confirm via the critical row:
+```
+max by (service_name) (consul_health_service_status{service_name="<name>",status="critical"})
+```
+A critical value of `1` is direct evidence the bare-metal service is down/unhealthy.
+
+## Healthy ≠ a root cause
+If the passing value is `1` and the critical value is `0`, the Consul service is
+**healthy** — do not manufacture a Consul cause. Either the incident is on a
+different axis (the host process is up but a dependency or the data plane is
+degraded — investigate that) or there is no live incident, in which case
+**conclude inconclusive**. Never confirm "Consul health failing" for a service
+whose passing row reads `1`.
+
+## To CONFIRM a failing Consul service (so the test verifies, not "absent")
+When you hypothesize the service is failing its Consul check, attach a checkable
+prediction whose metric the gather can reduce to a single value. Use the
+aggregated passing form so the evidence comes back as one number, not mixed rows:
+```json
+{"kind":"metric-threshold","metric":"consul_health_service_status","op":"<","value":1}
+```
+Then `query` it — the gather must run the AGGREGATED query
+`max by (service_name) (consul_health_service_status{service_name="<name>",status="passing"})`
+so it returns a single value. If that value is `< 1` (i.e. `0`), the `test` returns
+**satisfied** → you can `conclude` "bare-metal Consul service failing its health
+check". A bare `consul_health_service_status{...}` selector (no `status` filter, no
+aggregation) returns multiple 0/1 rows the keystone cannot reduce — it will always
+come back "couldn't verify". Always predict/gather the aggregated passing form.
+
+Once confirmed, deepen the cause: investigate the host process, its logs via the
+bare-metal logLabels, and any upstream dependency it relies on — but the failing
+Consul health check is itself a valid, grounded root cause to report.

skills/k8s-deployment-investigation.md

@@ -0,0 +1,61 @@
+---
+title: Kubernetes Workload Investigation
+services: []
+alerts: []
+appliesToServiceMetric: kube_
+healthySignal: '(kube_deployment_status_replicas_available{deployment="$service"} == kube_deployment_spec_replicas{deployment="$service"}) and (kube_deployment_spec_replicas{deployment="$service"} > 0)'
+failureSignal: 'kube_deployment_status_replicas_unavailable{deployment="$service"} > 0'
+failureCause: '$service has unavailable replicas (pods not ready) — the Kubernetes deployment is degraded.'
+identityHint: '$service is a Kubernetes workload (kube_deployment). Check its replica STATE first (available vs spec). A fully-available deployment is healthy — do not confirm CPU/OOM/readiness/pod faults without gathering the metric that shows the fault. spec==0 means scaled to zero (a valid root cause).'
+incompatibleClaims: '\bconsul\b|consul_health|bare-metal consul'
+tags:
+  - investigation
+  - rca
+  - kubernetes
+  - k8s
+scope:
+  - investigation
+---
+## When to use
+This runbook applies to Kubernetes workloads — identified by a `kube_deployment_*`
+(or `kube_statefulset_*` / `kube_pod_*`) metric. It is injected into an
+investigation only when the incident service is k8s-tracked and this skill is
+enabled. If the service is a bare-metal Consul service (it has a
+`consul_health_service_status` metric, no `kube_*`), this does not apply.
+
+## The health signal — check it FIRST
+Read the deployment's replica state before forming any pod-level hypothesis:
+```
+kube_deployment_spec_replicas{deployment="<name>"}
+kube_deployment_status_replicas_available{deployment="<name>"}
+kube_deployment_status_replicas_unavailable{deployment="<name>"}
+```
+- **available == spec, unavailable == 0** → the deployment is **healthy** on its
+  primary signal.
+- **spec == 0** → scaled to zero — a complete, valid root cause for unavailability
+  (predict `kube_deployment_status_replicas < 1`; it reads 0 → confirmed). With
+  zero replicas there are **no pods**, so pod-runtime causes (GPU/OOM/readiness)
+  do not apply.
+- **available < spec / unavailable > 0** → some pods are down — investigate pod
+  status, restarts, OOMKills, image pulls, scheduling.
+
+## Healthy ≠ a root cause (read this before confirming)
+If `available == spec` and `unavailable == 0`, the workload is **healthy** — do NOT
+manufacture a cause. In particular do NOT confirm "CPU throttling", "OOMKilled",
+"readiness failing", or "pods crashing" **unless you actually gathered the metric
+that shows it**:
+```
+rate(container_cpu_cfs_throttled_seconds_total{pod=~"<name>.*"}[5m])   # > 0 to claim throttling
+rate(kube_pod_container_status_restarts_total{pod=~"<name>.*"}[15m])   # > 0 to claim crashloop
+kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}    # to claim OOM
+```
+If those return no data / zero, the symptom isn't happening — **conclude
+inconclusive** ("deployment appears healthy; no incident found"), never a confirm.
+A confirm must name a fault for which a gathered observation crossed a threshold.
+
+## To CONFIRM a real k8s incident
+Attach a checkable prediction over the metric that shows the fault, e.g.
+`{"kind":"metric-threshold","metric":"kube_deployment_status_replicas_unavailable","op":">","value":0}`
+for unavailable replicas, or a restart-rate / throttle-rate threshold for the
+specific pod-level fault. The `test` move must come back **satisfied** on real,
+gathered evidence before you `conclude`.