ci: test-windows shard 1 reddens unrelated PRs — the parallel-scheduler contract's cleanup probe gets a 1s budget and fails closed on timeout

[DeusData]

Summary

test / test-windows (windows-latest, CLANG64, x86_64, 1/2) intermittently fails at Step 0i: parallel suite scheduler contract with:

FAIL: parallel scheduler infrastructure error: parallel scheduler cleanup failed:
hang_after_summary: suite 'hang_after_summary' leader exited leaving live descendants

It is the harness's own negative-control fixture failing, before any product test runs. It reddens PRs that cannot possibly have caused it, and shard-completeness plus ci-ok fail downstream, so the PR presents as three red checks.

Confirmed instances (both unrelated to the failure): #1179 (VB.NET grammar) and #1657 (a 3-file src/foundation/mem.* change). Neither touches Windows, the scheduler, or the parallel harness.

Root cause

scripts/run-test-wave.py:159-173, the Windows branch of terminate_process_tree, handles the leader-already-exited race by proving nothing is still parented to the dead PID:

if leader_exited:
    if windows_descendants(process.pid, kill_grace):
        raise RuntimeError(
            f"suite {active.name!r} leader exited leaving live descendants"
        )
    return

windows_descendants fails closed on any uncertainty (same file, just above):

except (OSError, subprocess.TimeoutExpired):
    return True  # cannot prove absence -> assume the worst
if completed.returncode != 0:
    return True

Failing closed is the right instinct for a cleanup check. The problem is the budget it gets: the probe is called with timeout=kill_grace, and the contract fixture runs with --kill-grace 1, i.e. a one-second budget for a Windows process-enumeration call. On a loaded hosted runner that probe times out or returns non-zero often enough to matter, and a timeout is indistinguishable here from a genuine surviving descendant.

The fixture makes this maximally likely: hang_after_summary prints its summary and then time.sleep(30) against --timeout 1, so the leader is always killed mid-sleep and the exited-leader branch is always the one taken. The negative control that exists to prove cleanup works is the case most exposed to the probe timing out.

The code comment directly above already records that this class of failure "is how a deliberately-hanging fixture suite reddened a release run" — so it has cost a release once already.

Why it looks like flake but is not random

Nothing about the product changes the outcome; only runner load does. A rerun usually passes, which is why it has been absorbed as noise rather than fixed. Under O9 that makes it a genuine defect: the verdict is currently a function of runner scheduling rather than of (code, test, platform, seed).

Suggested directions

1. Give the probe its own budget, independent of kill_grace. A cleanup-verification call and a grace period for a process to die are different quantities that happen to share a number today.
2. Distinguish "probe failed" from "descendants found." Failing closed is correct for the latter; for the former, retry briefly and only then fail — and say which happened in the message, since the current text asserts live descendants when it may only mean the probe timed out.
3. Do not let a harness-contract failure redden the product matrix without saying so — the message already names itself an "infrastructure error", so the shard could surface it distinctly from a test failure.

Reproduction

Any PR touching the Windows shard-1 matrix; it is load-dependent, not content-dependent. Instances: #1179 (33163242890-era run) and #1657 job 98825008892 (2026-08-28, failed 3m23s in, at Step 0i, with all preceding pass_N suites returning rc=0).