fix(deep): guard against the Consul 'not deployed in k8s' category error - #259
Closed
WZ wants to merge 3 commits into
Closed
fix(deep): guard against the Consul 'not deployed in k8s' category error#259WZ wants to merge 3 commits into
WZ wants to merge 3 commits into
Conversation
Trace of a bd-management run showed the keystone CONFIRMING a true-but-wrong cause:
'bd-management is not deployed in the cluster (no k8s pod exists)'. It's true (a
Consul service has no k8s pod) but it is NOT the root cause — and the run had
already gathered consul_health evidence proving the service is bare-metal Consul.
The skill tells the model not to do this, but it does ~half the time; a prompt
can't guarantee it.
Deterministic guard in the conclude path: if the run gathered any
consul_health_service_status evidence AND the confirmed cause asserts a missing
k8s deployment ('not deployed', 'no k8s pod', 'deployment missing/absent'), reject
it and make the agent confirm via the Consul health signal instead. Genuine k8s
incidents (e.g. agw-admin-ui's deleted namespace) gather no consul_health
evidence, so the guard never fires for them — proven by a control test.
tsc clean; 2 new tests (guard fires for Consul / allows genuine k8s); suite green.
Validation surfaced a second wrong-layer failure: investigating the idle minimax-m25-metrics-proxy, the orchestrator ruled out every real hypothesis, then chased its OWN query errors — 'datasource with UID loki not found' (MI350 has no Loki datasource configured) appearing as 404s in logs — and confirmed 'Grafana datasource misconfigured' as the root cause. That's a query-layer artifact, not why the service is unhealthy. Guard (same pattern as the Consul category-error guard, same conclude path): reject a confirmed cause that blames the observability tooling (mentions 'datasource' or 'grafana') UNLESS the incident service is itself an observability component (grafana/loki/prometheus/...). Forces the agent back to the service's own signals; an idle service with no real cause then exhausts/pauses honestly instead of inventing a tooling cause. tsc clean; 2 new tests (guard fires for a service / allows a grafana incident); suite green.
…-pause
Investigating minimax-m25-vllm-bench-4gpu (a k8s Deployment scaled to 0 replicas,
kube_deployment_spec_replicas=0), the orchestrator spent all ~30 moves chasing
Consul hypotheses ('is it a bare-metal Consul service / not registered in Consul /
health check unknown...'), never proposed the obvious 'deployment scaled to 0',
and operator-paused. Root cause: #256 injects the Consul runbook into EVERY
orchestrator run (always-on), which over-steers k8s services into Consul
tunnel-vision — a regression for the k8s case introduced by the Consul fixes.
Two parts:
- Skill: lead the 'When investigating' section with DETERMINE SERVICE TYPE FIRST —
query kube_deployment_status_replicas; if it returns data the service IS a k8s
Deployment (0 = scaled to 0 = a valid root cause, conclude it), and ONLY treat as
Consul when there's no kube_deployment metric. Stops the Consul tunnel-vision.
- Guard: the Consul category-error guard no longer fires when the run also saw
kube_deployment/kube_pod evidence — a real k8s Deployment's k8s-absence cause is
legitimate (only consul_health-AND-no-kube services are the bare-metal ones).
tsc clean; 1 new test (k8s service not blocked); suite green.
This was referenced Jun 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What the trace showed
A bd-management run had the keystone confirm a true-but-wrong cause: "bd-management is not deployed in the cluster (no k8s pod exists)". It's literally true (a Consul service has no k8s pod) but it's not the root cause — and the same run had already gathered
consul_healthevidence proving the service is bare-metal Consul. The #256 skill tells the model not to conclude this, but it does ~half the time. A prompt can't guarantee it.Deterministic guard
In the conclude path: if the run gathered any
consul_health_service_statusevidence and the confirmed cause asserts a missing k8s deployment (not deployed,no k8s pod,deployment missing/absent), reject it and make the agent confirm via the Consul health signal instead.Safe for genuine k8s incidents: agw-admin-ui's deleted-namespace cause gathers no
consul_healthevidence, so the guard never fires for it — pinned by a control test.Verification
Context
The consistency fix on top of the form→predict→gather→verify chain (#256/#257/#258). Makes the category error structurally impossible for Consul services. Feature still gated off.