Skip to content

fix(deep): aggregate Consul health metric in the evidence gather - #258

Closed
WZ wants to merge 1 commit into
mainfrom
fix/consul-gather-aggregation
Closed

fix(deep): aggregate Consul health metric in the evidence gather#258
WZ wants to merge 1 commit into
mainfrom
fix/consul-gather-aggregation

Conversation

@WZ

@WZ WZ commented Jun 10, 2026

Copy link
Copy Markdown
Owner

The final link in the chain

From the persisted traces of the #257 re-test: the orchestrator forms the right Consul hypothesis (#256) with a verifiable prediction (#257), but the evidence gather queried the bare consul_health_service_status metric — which returns one series per (node × status), mixed 0/1 — so the extractor produced zero usable observations and the keystone returned absent:

proposed a cause: impala-statestore is unhealthy …
gathered evidence  … → +0 observations          ← bare metric, no usable value
couldn't verify    impala-statestore is unhealthy …

(bd-management happened to confirm via Consul; impala kept coming up empty → wall-clock/exhausted.)

Fix

planPredictionQuery appends a Consul-aware hint to the metric-threshold gather prompt: query it aggregated for the service in the hypothesis —
max by (service_name) (consul_health_service_status{service_name="X",status="passing"}) → one clean value (1 passing / 0 failing) the keystone can verify → confirm.

Triggers only when the metric is consul_health_service_status (no false positives for ordinary metrics).

Verification

  • tsc clean · 2 new tests (hint present for consul / absent for ordinary) · full suite green
  • Live re-test pending — expect impala to now confirm the Consul-health cause consistently (matching bd-management)

Context

Closes the form → predict → gather → verify chain for bare-metal Consul incidents. The iteration that should make impala pass consistently. Feature still gated off.

Final link in the chain: the orchestrator now forms the right Consul hypothesis
(#256) with a verifiable prediction (#257), but the evidence gather queried the
BARE consul_health_service_status metric — which returns one series per
(node x status), mixed 0/1 — so the extractor produced zero usable observations
and the keystone returned 'absent' ('couldn't verify'). Trace: impala runs kept
proposing 'impala-statestore unhealthy' then gathering +0 observations.

Fix: planPredictionQuery now appends a Consul-aware hint to the metric-threshold
gather prompt — query it AGGREGATED for the service in the hypothesis:
max by (service_name) (consul_health_service_status{service_name="X",status="passing"})
→ one clean value (1 passing / 0 failing) the keystone can verify → confirm.

Only triggers when the metric is consul_health_service_status (no false positives
for ordinary metrics). tsc clean; 2 new tests + full suite green.

Closes the form→predict→gather→verify chain for bare-metal Consul incidents.
@WZ

WZ commented Jun 11, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #261, which combines #258 + #259 + #260 into a single inc-7 accuracy PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant