[Visual Test] Label preflight regressed again: unpaginated gh label list still misses run-visual-test on run #20

[MichaelFisher1997]

Workflow run

https://github.com/OpenStaticFish/ZigCraft/actions/runs/29312935256

Trigger: scheduled nightly, commit 16527e3 (feat: add generated preloaded menu preview #915), dev branch.

Symptom

The visual-test workflow fails at step 8 ("Ensure visual-test label exists") and never reaches step 10 ("Run menu screenshot capture"). No screenshot.png is produced and no build-output.log is written. Every downstream step (Setup Lavapipe Vulkan, Run menu screenshot capture, Compare against golden image, Upload screenshot artifact, Upload build log artifact) is skipped. The visual-regression gate is effectively dead until this is fixed.

This is the third consecutive nightly run to fail at this same preflight. #916 (runs #17 / #18) reported it first; #931 (run #19) reported the regression one scheduled run later; this issue reports run #20. The fix proposed in both prior issues has not landed, so the workflow has been stuck at step 8 every night since 2026-07-12.

Why build-output.log is missing

The diagnosis workspace does not contain build-output.log. That is consistent with the failure: build-output.log is only created by the run-with-log composite action used in the "Run menu screenshot capture" step (.github/workflows/visual-test.yml:76-81). Because step 10 was skipped, the tee never ran and no log file was written.

The only artifact from the runner is weston.log, which shows the compositor started normally and was killed 1.4 seconds later by the post-failure cleanup step:

Zig 0.16.0 CI graphics environment
Compiler: 0.16.0
Date: 2026-07-14 UTC
[06:58:53.865] weston 15.0.1
               https://wayland.freedesktop.org
               ...
[06:58:53.865] Command line: weston --socket=headless --backend=headless-backend.so --width=1280 --height=720
...
[06:58:53.870] Output 'headless' enabled with head(s) headless
...
[06:58:55.299] caught signal 15
[06:58:55.299] no-op renderer SHM seed: 0

Signal 15 (SIGTERM) at 06:58:55 lines up with step 8 completing in a failed state shortly after 06:58:55, and the always-run "Stop headless Wayland compositor" cleanup step firing immediately afterward. The game binary was never executed, so there is nothing to diagnose in the engine, screenshot path, Lavapipe, or shader pipelines.

Job step status (run #20, attempt 1)

#	Step	Status
1	Set up job	success
2	Run actions/checkout@v4	success
3	Setup Nix	success
4	Start headless Wayland compositor	success
5	Prepare opencode cache	success
6	Load visual verification prompt	success
7	Load visual diagnosis prompt	success
8	Ensure visual-test label exists	failure
9	Setup Lavapipe Vulkan	skipped
10	Run menu screenshot capture	skipped
11	Check screenshot exists	success (exists=false)
12	Compare against golden image	skipped
13	Upload screenshot artifact	skipped
14	Check build log exists	success (exists=false)
15	Upload build log artifact	skipped
16	Stop headless Wayland compositor	success
17	Run opencode visual verification	skipped
18	Run opencode failure diagnosis	in_progress (running opencode here)

Step 8 fails before any of the engine-level code (screenshot.zig, vulkan_swapchain.zig, HomeScreen, Lavapipe, shaders) is exercised, so this is purely a workflow-config bug.

Root cause

The pre-flight existence check in .github/workflows/visual-test.yml:55-68 is buggy. gh label list (without --paginate or a higher --limit) returns only the first 30 labels, sorted alphabetically. The ZigCraft repo currently has 41 labels, so labels that sort past position 30 are invisible to the check. Alphabetical sort rank of the two labels in question, in the current label set:

Rank	Label
1	audit-b1
2	audit-b2
...	...
30	perf/bandwidth
31	perf/gpu-compute
32	perf/rendering
33	production-readiness
34	question
35	run-visual-test
...	...
38	visual-test
...	...

Both run-visual-test (rank 35) and visual-test (rank 38) are past the 30-label page cutoff. Both gh label list --json name --jq '.[].name' calls at .github/workflows/visual-test.yml:57 and :62 therefore return an empty match, both gh label create calls fire, and both fail because the labels already exist:

$ gh label list --repo OpenStaticFish/ZigCraft --json name --jq '.[].name' | wc -l
30
$ gh label list --repo OpenStaticFish/ZigCraft --limit 100 --json name --jq '.[].name' | wc -l
41

gh label create exits non-zero on duplicate. The step aborts on the first gh label create failure (the script uses set -e/pipefail semantics implicitly via run:), and the workflow proceeds to its failure() path.

Note: #916 and #931 were written when the label set had different sort positions; on run #17 the cutoff hid only run-visual-test and visual-test happened to land inside the first 30. Since then the label inventory has grown and now hides both. The bug is the same regardless — the unpaginated gh label list is the culprit.

Origin

• .github/workflows/visual-test.yml:55-68 — Ensure visual-test label exists step.
• Specifically .github/workflows/visual-test.yml:57 and .github/workflows/visual-test.yml:62 — the unpaginated gh label list --json name --jq '.[].name' calls.

The downstream game code paths were not exercised at all on this run, so the screenshot pipeline (modules/engine-graphics/src/vulkan/screenshot.zig), the headless swapchain (modules/engine-graphics/src/vulkan_swapchain.zig:119-166), and HomeScreen rendering (modules/game-ui/src/screens/home.zig) cannot be blamed for this failure. If the preflight is fixed the next run should either succeed or surface a different engine-level failure worth investigating.

Suggested fix

Pick one of the following (in increasing order of robustness). These match what #916 already recommends; option 1 is the smallest delta that preserves the existing "skip if present" intent:

1. Page the label list (minimal change)

Add --paginate (or --limit 100) to both gh label list invocations:

- name: Ensure visual-test label exists
  run: |
    if ! gh label list --paginate --json name --jq '.[].name' | grep -q '^visual-test$'; then
      gh label create "visual-test" \
        --description "Issues from automated visual regression tests" \
        --color "E06C75"
    fi
    if ! gh label list --paginate --json name --jq '.[].name' | grep -q '^run-visual-test$'; then
      gh label create "run-visual-test" \
        --description "Run deterministic visual regression workflow on a PR" \
        --color "E06C75"
    fi

2. Make the create idempotent

- name: Ensure visual-test label exists
  run: |
    gh label create "visual-test" \
      --description "Issues from automated visual regression tests" \
      --color "E06C75" \
      --force
    gh label create "run-visual-test" \
      --description "Run deterministic visual regression workflow on a PR" \
      --color "E06C75" \
      --force

3. Check a single label at a time

- name: Ensure visual-test label exists
  run: |
    for spec in 'visual-test|E06C75|Issues from automated visual regression tests' \
                'run-visual-test|E06C75|Run deterministic visual regression workflow on a PR'; do
      name="${spec%%|*}"
      rest="${spec#*|}"
      color="${rest%%|*}"
      desc="${rest#*|}"
      if gh label list --search "^${name}$" --json name --jq '.[].name' | grep -qx "${name}"; then
        echo "label ${name} already exists"
      else
        gh label create "${name}" --description "${desc}" --color "${color}"
      fi
    done

Any of these is sufficient. Option 1 is preferred because it preserves the "skip if present" intent of the original script with the smallest patch surface.

Recommended next steps

1. Open a hotfix/* PR against dev with Option 1 (or any of the alternatives) on both calls at visual-test.yml:57 and :62. Two-line YAML change; doesn't need a feature branch.
2. Once the preflight lands and the next scheduled run reaches step 10, watch for an actual engine-level screenshot/diff result before declaring victory — that is the first time the gate will have executed end-to-end.
3. Close [Visual Test] Label preflight fails: gh label list only returns first 30 labels, run-visual-test is on page 2 #916 and [Visual Test] Label preflight regressed: gh label list pagination skips run-visual-test on run #19 #931 as duplicates of whichever lands first (recommend closing [Visual Test] Label preflight fails: gh label list only returns first 30 labels, run-visual-test is on page 2 #916 and [Visual Test] Label preflight regressed: gh label list pagination skips run-visual-test on run #19 #931 as dups of this one, since this report cross-references all three runs). Then re-open / re-investigate [Visual Test] PPM screenshot capture is unsupported #913 and [Visual Test] Menu screenshot RMSE 0.0437 exceeds 0.015 tolerance after g_pipeline inherits alphaToCoverageEnable #911 only if a real engine failure surfaces after the preflight is reachable.

Related

• [Visual Test] Label preflight fails: gh label list only returns first 30 labels, run-visual-test is on page 2 #916 — first report of this exact preflight failure (runs Refactor: Split main.zig into modular game components #17, Refactor: Decompose App and Fix Vulkan Parity/SOLID issues #18). Still open.
• [Visual Test] Label preflight regressed: gh label list pagination skips run-visual-test on run #19 #931 — second report (run 🔍 Conduct Comprehensive Code Audit - Architecture, Performance, SOLID, and Quality Standards #19), same root cause. Still open.
• [Visual Test] PPM screenshot capture is unsupported #913, [Visual Test] Menu screenshot RMSE 0.0437 exceeds 0.015 tolerance after g_pipeline inherits alphaToCoverageEnable #911 — earlier engine-level visual-test failures that were attributed to engine code only because the preflight aborted before the build ever ran; they should be re-evaluated once this gate is actually reachable.

PR search for paginate / label list / visual-test against dev: no in-flight fix branch. Nothing is currently racing this. The regression has now persisted across three scheduled runs without a fix.

[github-actions]

Confirmed bug — classification: Bug (CI workflow config), not engine code.

Verification of the report

I verified the root cause directly against the repo:

• .github/workflows/visual-test.yml:57 and :62 use unpaginated gh label list --json name --jq '.[].name', exactly as reported.
• gh label list --repo OpenStaticFish/ZigCraft --limit 100 --json name --jq '.[].name' | wc -l -> 41
• gh label list --repo OpenStaticFish/ZigCraft --json name --jq '.[].name' | wc -l -> 30

The 30-label default page does in fact hide both run-visual-test and visual-test from the preflight, so the duplicate-create failure path triggers on every run. The engine code paths (screenshot.zig, vulkan_swapchain.zig, home.zig) are not implicated — step 8 fails before any of them can run, matching the report.

No in-flight fix

PR search for paginate / label list / visual-test against dev returns nothing. The head branch list of open PRs contains no branch targeting this file. Nothing is racing the fix, so a small hotfix PR is safe to land immediately.

Scope

This is appropriately scoped — it is a single, well-bounded 2-line YAML change at visual-test.yml:57 and :62. No need to split into sub-issues. However, since the regression has now blocked three scheduled runs (#916, #931, this issue), I would suggest filing it as a hotfix/* PR per the AGENTS.md workflow rules rather than feature/*, and targeting dev (not main).

Recommended implementation

Agree with Option 1 from the issue as the smallest correct fix:

- name: Ensure visual-test label exists
  run: |
    if ! gh label list --paginate --json name --jq '.[].name' | grep -q '^visual-test$'; then
      gh label create "visual-test" \
        --description "Issues from automated visual regression tests" \
        --color "E06C75"
    fi
    if ! gh label list --paginate --json name --jq '.[].name' | grep -q '^run-visual-test$'; then
      gh label create "run-visual-test" \
        --description "Run deterministic visual regression workflow on a PR" \
        --color "E06C75"
    fi

Two-line delta, preserves the "skip if present" intent, and is robust to the label inventory growing past 30 (or 100) in the future.

Option 2 (--force) is also valid and arguably simpler, but changes semantics from "skip if present" to "always upsert color/description". That may or may not be desired, so Option 1 is safer for a minimal hotfix.

Duplicates and follow-ups

• [Visual Test] Label preflight fails: gh label list only returns first 30 labels, run-visual-test is on page 2 #916 and [Visual Test] Label preflight regressed: gh label list pagination skips run-visual-test on run #19 #931 are duplicates of this report (same root cause, same preflight, same fix). They should be closed as dups once any of the three lands.
• [Visual Test] PPM screenshot capture is unsupported #913 and [Visual Test] Menu screenshot RMSE 0.0437 exceeds 0.015 tolerance after g_pipeline inherits alphaToCoverageEnable #911 were attributed to engine code purely because the preflight aborted before the build ran. They should not be re-investigated until the preflight is fixed and the next scheduled run actually reaches step 10 — at that point, if the build/screenshot pipeline fails, a fresh engine-level investigation is warranted.

Verification path after the hotfix lands

1. Confirm the next nightly run reaches step 10 ("Run menu screenshot capture") and produces screenshot.png + build-output.log.
2. If step 10 then fails, that's the first real signal in this gate's lifetime and warrants a fresh issue.
3. If step 10 succeeds but "Compare against golden image" fails, that is a legitimate visual regression and should be investigated against the recent change in commit 16527e3 ("feat: add generated preloaded menu preview feat: add generated preloaded menu preview #915").

No additional reproduction info is needed from the reporter — the failure mode, root cause, fix, and verification path are all already pinned down.

[github-actions]

Comment posted on issue #935: #935 (comment)

Verified the report directly: unpaginated gh label list returns 30 of 41 labels, hiding both run-visual-test and visual-test and causing the duplicate-create failure at step 8. Confirmed no in-flight PR addresses this, validated the proposed Option 1 fix, and recommended hotfix/* against dev per AGENTS.md.

opencode session  |  github run