ci: split self-hosted E2E lanes into their own workflow (EAI-7548) - #193
Open
fredespi wants to merge 1 commit into
Open
ci: split self-hosted E2E lanes into their own workflow (EAI-7548)#193fredespi wants to merge 1 commit into
fredespi wants to merge 1 commit into
Conversation
A job queued on an OFFLINE self-hosted runner cannot be cancelled by GitHub. While the self-hosted GPU E2E lanes shared ci.yml's concurrency group, a superseded run whose GPU job was queued on an offline runner kept holding the group, so the newer run's merge-required (GitHub-hosted) checks sat pending with zero jobs and had to be force-cancelled by hand (PR #138). timeout-minutes cannot fix this: the job-timeout timer only counts running time, never the queued/"waiting for a runner" phase, so it never reaps the stuck job. The only robust fix is structural separation. Move e2e-gpu, e2e-gpu-strix-ubuntu, and e2e-gpu-strix-windows into a new e2e-selfhosted.yml with its own concurrency group. ci.yml's shared group now holds only cancellable GitHub-hosted jobs, so supersession works cleanly and the required checks always start. The new workflow carries its own trimmed changes gate (cross-workflow needs is unavailable) and its own consolidated report (named distinctly to avoid colliding with ci.yml's required report). Teach xtask e2e-report discovery to handle download-artifact@v8's single-artifact layout: when exactly one artifact matches, v8 extracts it into the artifacts-dir root (no per-artifact subdir), which each report job now hits after the split. Discovery labels a root-level report from its platform.json slug, or a neutral Unknown/Unknown identity when the sidecar is absent (a GPU run can error before writing it) so a hardware failure is never misattributed. Add dependency-free contract tests (workflow_contract.rs) that fail if a self-hosted lane reappears in ci.yml or the two workflows lose their distinct concurrency namespaces, plus discovery and descriptor regression tests. Update docs/ci-hardware-testing.md and the e2e-cucumber README for the split. Note: the four self-hosted checks remain in the branch's required list, so an offline runner can still block a merge via a missing required check until they are removed from that list (a separate branch-protection change). Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A job queued on an offline self-hosted runner cannot be cancelled by GitHub. While the self-hosted GPU E2E lanes shared
ci.yml's concurrency group, a superseded run whose GPU job was queued on an offline runner kept holding the group — so the newer run's merge-required (GitHub-hosted) checks sat pending with zero jobs and had to be force-cancelled by hand (seen on #138).timeout-minutescannot fix this: the job-timeout timer only counts running time, never the queued/"waiting for a runner" phase, so it never reaps the stuck job. The only robust fix is structural separation.What changed
e2e-selfhosted.ymlholds the three self-hosted GPU lanes (e2e-gpu,e2e-gpu-strix-ubuntu,e2e-gpu-strix-windows) plus their own consolidated report, with its own concurrency group. An offline runner can now only ever stall this workflow's own supersession, neverci.yml's required checks.ci.ymlkeeps the GitHub-hosted mocke2elane and its requiredE2E consolidated report. Its shared concurrency group now holds only cancellable hosted jobs, so supersession works cleanly and the required checks always start.changesgate (cross-workflowneedsis unavailable) and builds therocmbinary itself (nobuild-and-testdependency);ci.yml's requiredbuild-and-test+ mocke2eremain the authoritative pre-merge build gate.xtask e2e-reportdiscovery now handlesdownload-artifact@v8's single-artifact layout: when exactly one artifact matches, v8 extracts it into the artifacts-dir root (no per-artifact subdir), which each report job now hits after the split. A root-level report is labeled from itsplatform.jsonslug, or a neutralUnknown/Unknownidentity when the sidecar is absent (a GPU run can error before writing it) so a hardware failure is never misattributed.xtask/src/workflow_contract.rs(dependency-free) fails if a self-hosted lane reappears inci.ymlor the two workflows lose their distinct concurrency namespaces; plus discovery + descriptor tests ine2e-report.docs/ci-hardware-testing.mdand the e2e-cucumber README for the split.Note for reviewers
The four self-hosted checks remain in the branch's required-status-check list, so an offline runner can still block a merge via a missing required check until they are removed from that list — a separate branch-protection change, out of scope for this workflow change.
Test plan
clippy --workspace --all-targets --exclude e2e-cucumber -- -D warningsclean (Linux)workflow_contract+e2e-reportdiscovery/descriptor tests pass