[Visual Test] Label preflight fails: gh label list only returns first 30 labels, run-visual-test is on page 2

[MichaelFisher1997]

Workflow run

https://github.com/OpenStaticFish/ZigCraft/actions/runs/29183938238

Symptom

The visual-test workflow fails at step 8 ("Ensure visual-test label exists") and never reaches step 10 ("Run menu screenshot capture"). As a result, no screenshot.png is produced and no build-output.log is written — every downstream step is skipped. This is the same failure observed on the previous scheduled run (#17, https://github.com/OpenStaticFish/ZigCraft/actions/runs/29143688205) and re-occurred on the run that produced this issue (#18).

Relevant error output

From 0_visual-test.txt for run #17 (the failure is byte-for-byte identical on run #18):

2026-07-11T06:59:02.2188083Z if ! gh label list --json name --jq '.[].name' | grep -q '^run-visual-test$'; then
2026-07-11T06:59:02.2188445Z   gh label create "run-visual-test" \
2026-07-11T06:59:02.2188798Z     --description "Run deterministic visual regression workflow on a PR" \
2026-07-11T06:59:02.2189127Z     --color "E06C75"
2026-07-11T06:59:02.2189340Z fi
…
2026-07-11T06:59:03.0186758Z label with name "run-visual-test" already exists; use `--force` to update its color and description
2026-07-11T06:59:03.0197803Z ##[error]Process completed with exit code 1.

Job step status for run #18 (current run):

#	Step	Status
8	Ensure visual-test label exists	failure
9	Setup Lavapipe Vulkan	skipped
10	Run menu screenshot capture	skipped
11	Check screenshot exists	success (exists=false)
12-17	downstream steps	skipped
18	Run opencode failure diagnosis	in_progress

Root cause

The pre-flight existence check in .github/workflows/visual-test.yml:55-68 is buggy:

- name: Ensure visual-test label exists
  run: |
    if ! gh label list --json name --jq '.[].name' | grep -q '^visual-test$'; then
      gh label create "visual-test" \
        --description "Issues from automated visual regression tests" \
        --color "E06C75"
    fi
    if ! gh label list --json name --jq '.[].name' | grep -q '^run-visual-test$'; then
      gh label create "run-visual-test" \
        --description "Run deterministic visual regression workflow on a PR" \
        --color "E06C75"
    fi

gh label list (without --paginate or a higher --limit) returns only the first 30 labels, sorted alphabetically. This repo now has 41 labels. Both target labels fall outside the first page when sorted alphabetically:

$ gh label list --repo OpenStaticFish/ZigCraft --json name --jq '.[].name' | nl | tail -15
    16  world
    17  shaders
    18  hotfix
    19  automated-audit
    20  automated-test
    21  visual-test          # <-- last label that IS returned by default
    22  batch-1
    23  batch-2
    …
    30  perf/gpu-compute
# run-visual-test, perf/bandwidth, … are not in this output

$ gh label list --repo OpenStaticFish/ZigCraft --json name --jq '.[].name' | wc -l
30

Because run-visual-test is past position 30 in the default page, grep -q '^run-visual-test$' exits 1, the if ! inverts that to 0 (true), and gh label create "run-visual-test" runs — and gh label create errors out because the label already exists:

label with name "run-visual-test" already exists; use `--force` to update its color and description

visual-test happens to land at position 21, inside the default page, so that check is currently lucky. The bug is the same for both checks — only run-visual-test is failing today because it sorts past the 30-item cutoff. As more labels are added to the repo, visual-test will also drop off the first page and start failing.

The check has been broken since the repo crossed 30 labels. It never had any effect on the visual-test job — it has always aborted before step 10 even ran — which is why no build-output.log exists in the diagnosis workspace for runs #17 and #18: the build was never invoked.

(Note: issue #913 from run #17 was filed by the same diagnosis flow but misattributed the failure to "PPM screenshot capture is unsupported" because the build log was missing. The actual failure on that run — and on run #18 — is the label-creation error above. #913 should be closed once this issue is fixed.)

File / function where the failure originates

• File: .github/workflows/visual-test.yml
• Step: Ensure visual-test label exists (lines 55-68)
• Failing command: gh label create "run-visual-test" invoked on line 63, reached because the guard on line 62 returns the wrong answer when paginated.

Suggested fix

Either fetch all labels so the guard sees the full list, or make the create itself idempotent. Either of the following edits to .github/workflows/visual-test.yml resolves the failure:

Option A — paginate the list query (smallest change)

- name: Ensure visual-test label exists
  run: |
    if ! gh label list --paginate --json name --jq '.[].name' | grep -q '^visual-test$'; then
      gh label create "visual-test" \
        --description "Issues from automated visual regression tests" \
        --color "E06C75"
    fi
    if ! gh label list --paginate --json name --jq '.[].name' | grep -q '^run-visual-test$'; then
      gh label create "run-visual-test" \
        --description "Run deterministic visual regression workflow on a PR" \
        --color "E06C75"
    fi
  env:
    GH_TOKEN: ${{ secrets.OPENCODE_PAT }}

Option B — gh label edit --force so it works whether the label exists or not

- name: Ensure visual-test label exists
  run: |
    gh label create "visual-test" \
      --description "Issues from automated visual regression tests" \
      --color "E06C75" || \
      gh label edit "visual-test" \
        --description "Issues from automated visual regression tests" \
        --color "E06C75"
    gh label create "run-visual-test" \
      --description "Run deterministic visual regression workflow on a PR" \
      --color "E06C75" || \
      gh label edit "run-visual-test" \
        --description "Run deterministic visual regression workflow on a PR" \
        --color "E06C75"
  env:
    GH_TOKEN: ${{ secrets.OPENCODE_PAT }}

Option A keeps the original "skip if already present" intent and just makes the check correct. Option B also re-syncs the description/color on every run, which is slightly more invasive but never fails.

Either way, once this step succeeds, the workflow will reach step 10 and the real screenshot.png capture / golden-diff CI can finally run, so we can verify whether the underlying graphics stack is healthy.

[github-actions]

Classification: Bug.

Validated the report against the repo and current gh behavior:

• .github/workflows/visual-test.yml:57 and .github/workflows/visual-test.yml:62 call gh label list --json name without raising the default limit.
• In this repo, gh label list --json name --jq 'length' returns 30, while gh label list --limit 1000 --json name --jq 'length' returns 41.
• run-visual-test is not present in the default 30-label result, but is present when the limit is raised, so the guard incorrectly falls through to .github/workflows/visual-test.yml:63 and gh label create fails because the label already exists.

No more repro data is needed; the root cause is clear and localized to the label preflight step. This is not too broad and should be a small CI hotfix.

Recommended fix: keep the existing guard but make the label query complete, e.g. use gh label list --limit 1000 --json name --jq '.[].name' for both checks. Note that the installed gh label list help exposes --limit but not --paginate, so --limit 1000 or an explicit paginated gh api call is safer than gh label list --paginate.

Related: #913 looks like a downstream/misattributed diagnosis from the same failure mode, since this workflow currently requests screenshot.png at .github/workflows/visual-test.yml:81 and never reached that step in the failed runs. Open PR #896 appears to be visual-test output/artifacts only, not a fix for this preflight failure.

[github-actions]

Comment posted: #916 (comment)

opencode session  |  github run