test(e2e): retry the Observe tab key until the dashboard acts on it

[rominf]

Summary

Five dashboard e2e scenarios (dash-loading-service-status, dash-managed-service-metrics, dash-managed-service-visible, and both eai-7960-*) can fail ~30s after they start, in an assertion about the Observe view — while the captured screen shows the dashboard still sitting on Home, ● Home active and the Observe chip untouched.

Cause. Each of those scenarios launches the TUI and comes straight to "the user opens the Observe view", with no assertion in between to prove the dashboard is reading input yet. TuiSession::send writes into the pseudo-terminal regardless, so a 4 written before the crossterm event loop is reading can be consumed by whatever holds the terminal at that moment. Nothing retries it, the dashboard never leaves Home, and the failure only surfaces one step later when the next wait_for_screen burns its full 30s timeout on a view the scenario never navigated to.

Fix. Adds TuiSession::send_until(bytes, marker, timeout), which re-sends the key every 500ms until the screen shows the marker, and switches the Observe step to send_until("4", "● Observe", DEFAULT_TIMEOUT). The step now depends on the dashboard having acted on the key rather than on the key having been written at a moment the dashboard happened to be ready.

Why re-sending is safe here, and only here. The helper is documented as safe only for idempotent keys — a tab jump, not a toggle. 4 maps unconditionally to "switch to Observe", so pressing it again while already there is a no-op; a ? (help toggle) or : (palette) would flip state on every repeat and must not use this helper. The doc states that constraint, including that copies of the key may still be queued when the call returns.

Terminal failures are not retried: if the child exits or the PTY reader thread dies, send_until returns that error immediately. wait_for_screen consumes the reader's panic message, so retrying past it would destroy the diagnostic and then report a generic timeout instead of the cause it already had.

Verification

Reproduced deterministically by pointing ROCM_CLI_BINARY at a wrapper that holds the terminal in non-canonical mode for two seconds, swallowing everything written to it, before exec'ing the real binary:

#!/bin/bash
stty -icanon -echo min 0 time 0
deadline=$((SECONDS + 2))
while [ "$SECONDS" -lt "$deadline" ]; do
  IFS= read -r -N 1 -t 0.05 ch && printf 'SWALLOWED:%q\n' "$ch" >&2
done
exec /path/to/real/rocm "$@"

• Before: the wrapper swallows the single 4. dash-managed-service-metrics fails 32s later with TTFT metrics did not appear: timed out after 30s waiting for "50ms", and the captured screen shows the dashboard still on Home.
• After: the wrapper swallows four 4s during its window; the first resend after exec lands, and the scenario passes.

Also run locally:

• all 9 dash.feature scenarios green under normal conditions (no wrapper)
• cargo fmt --all --check
• cargo clippy --locked --workspace --all-targets -- -D warnings
• cargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings — the e2e target sets test = false, so --all-targets skips it and it must be linted (and compiled) explicitly

Caveat, stated plainly: the reproduction is synthetic. It proves the failure mode is real and that the change removes it, but it does not prove this is the mechanism behind the CI flake — that remains unconfirmed. The change is defensive.

Scope

Split out of #165, where it was unrelated to that PR's subject (atomic downloads).

Two sibling steps have the same structural exposure and are deliberately left alone here: open_dashboard_help (?) and open_command_palette (:) also send a key straight after launch. Both keys are toggles, so send_until cannot be used for them — if they start flaking, the right fix there is a readiness wait before the key, not a resend.

Risk: low. Test harness only, no product code.

---

• If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. — no rows reference these scenarios.

[rominf]

CI status: all 16 required checks are green, including the blocking mock E2E tests lane and windows-build-and-test.

E2E tests (GPU) (advisory, self-hosted) is red with a single unexpected failure, serve-hf-checkpoint-inference — rocm serve failed on the "canonical Hugging Face GGUF checkpoint" step. It is pre-existing and unrelated to this PR: the same lone failure, with the same reconciliation line (5 xfail …, 1 unexpected failure(s)), appears on a run of main itself that started before this branch existed (run 31799869072). Nothing in this diff touches serving — it changes only tui_driver.rs and dash_steps.rs.

The dashboard scenarios this PR is about all passed on that GPU runner, including every step that goes through the changed code path:

Scenario: 3 - The dashboard reports a model that is still loading as loading
 ✔  And the user opens the Observe view
Scenario: 4 - Observe displays metrics from a managed model
 ✔  And the user opens the Observe view
Scenario: 7 - A managed model is visible in the dashboard
 ✔  And the user opens the Observe view
Scenario: 8 - Gen throughput stays visible for the validity window after a scrape failure
 ✔  And the user opens the Observe view
Scenario: 9 - Gen throughput expires after the validity window following sustained failure
 ✔  And the user opens the Observe view

The two Strix Halo lanes are still queued behind other branches' runs on the shared self-hosted runners; I'll follow up here if either reports anything attributable to this change.