Skip to content

ci: split self-hosted E2E lanes into their own workflow (EAI-7548) - #193

Open
fredespi wants to merge 1 commit into
mainfrom
fix-ci-selfhosted-lane-timeout
Open

ci: split self-hosted E2E lanes into their own workflow (EAI-7548)#193
fredespi wants to merge 1 commit into
mainfrom
fix-ci-selfhosted-lane-timeout

Conversation

@fredespi

@fredespi fredespi commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

A job queued on an offline self-hosted runner cannot be cancelled by GitHub. While the self-hosted GPU E2E lanes shared ci.yml's concurrency group, a superseded run whose GPU job was queued on an offline runner kept holding the group — so the newer run's merge-required (GitHub-hosted) checks sat pending with zero jobs and had to be force-cancelled by hand (seen on #138).

timeout-minutes cannot fix this: the job-timeout timer only counts running time, never the queued/"waiting for a runner" phase, so it never reaps the stuck job. The only robust fix is structural separation.

What changed

  • New workflow e2e-selfhosted.yml holds the three self-hosted GPU lanes (e2e-gpu, e2e-gpu-strix-ubuntu, e2e-gpu-strix-windows) plus their own consolidated report, with its own concurrency group. An offline runner can now only ever stall this workflow's own supersession, never ci.yml's required checks.
  • ci.yml keeps the GitHub-hosted mock e2e lane and its required E2E consolidated report. Its shared concurrency group now holds only cancellable hosted jobs, so supersession works cleanly and the required checks always start.
  • The self-hosted workflow carries its own trimmed changes gate (cross-workflow needs is unavailable) and builds the rocm binary itself (no build-and-test dependency); ci.yml's required build-and-test + mock e2e remain the authoritative pre-merge build gate.
  • xtask e2e-report discovery now handles download-artifact@v8's single-artifact layout: when exactly one artifact matches, v8 extracts it into the artifacts-dir root (no per-artifact subdir), which each report job now hits after the split. A root-level report is labeled from its platform.json slug, or a neutral Unknown/Unknown identity when the sidecar is absent (a GPU run can error before writing it) so a hardware failure is never misattributed.
  • New regression tests: xtask/src/workflow_contract.rs (dependency-free) fails if a self-hosted lane reappears in ci.yml or the two workflows lose their distinct concurrency namespaces; plus discovery + descriptor tests in e2e-report.
  • Updated docs/ci-hardware-testing.md and the e2e-cucumber README for the split.

Note for reviewers

The four self-hosted checks remain in the branch's required-status-check list, so an offline runner can still block a merge via a missing required check until they are removed from that list — a separate branch-protection change, out of scope for this workflow change.

Test plan

  • clippy --workspace --all-targets --exclude e2e-cucumber -- -D warnings clean (Linux)
  • Full workspace tests + e2e-cucumber lib tests pass (Linux)
  • New workflow_contract + e2e-report discovery/descriptor tests pass
  • Both workflow files parse; all five required check names still produced, one workflow each

A job queued on an OFFLINE self-hosted runner cannot be cancelled by
GitHub. While the self-hosted GPU E2E lanes shared ci.yml's concurrency
group, a superseded run whose GPU job was queued on an offline runner kept
holding the group, so the newer run's merge-required (GitHub-hosted) checks
sat pending with zero jobs and had to be force-cancelled by hand (PR #138).

timeout-minutes cannot fix this: the job-timeout timer only counts running
time, never the queued/"waiting for a runner" phase, so it never reaps the
stuck job. The only robust fix is structural separation.

Move e2e-gpu, e2e-gpu-strix-ubuntu, and e2e-gpu-strix-windows into a new
e2e-selfhosted.yml with its own concurrency group. ci.yml's shared group now
holds only cancellable GitHub-hosted jobs, so supersession works cleanly and
the required checks always start. The new workflow carries its own trimmed
changes gate (cross-workflow needs is unavailable) and its own consolidated
report (named distinctly to avoid colliding with ci.yml's required report).

Teach xtask e2e-report discovery to handle download-artifact@v8's
single-artifact layout: when exactly one artifact matches, v8 extracts it into
the artifacts-dir root (no per-artifact subdir), which each report job now hits
after the split. Discovery labels a root-level report from its platform.json
slug, or a neutral Unknown/Unknown identity when the sidecar is absent (a GPU
run can error before writing it) so a hardware failure is never misattributed.

Add dependency-free contract tests (workflow_contract.rs) that fail if a
self-hosted lane reappears in ci.yml or the two workflows lose their distinct
concurrency namespaces, plus discovery and descriptor regression tests. Update
docs/ci-hardware-testing.md and the e2e-cucumber README for the split.

Note: the four self-hosted checks remain in the branch's required list, so an
offline runner can still block a merge via a missing required check until they
are removed from that list (a separate branch-protection change).

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi
fredespi requested a review from a team as a code owner August 7, 2026 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant