Persist Slurm run state, retry shards, and collect winners

[nabinchha]

Priority Level

High

Task Summary

Implement durable run, shard, attempt, readiness, winner, retry, and collection behavior so every public control operation works from persisted records and normalized scheduler observations.

Technical Details & Implementation Plan

• Persist immutable run intent and resolved-plan references plus stable shard identities and record ranges.
• Create monotonically numbered attempts with scheduler array/job identities and atomic lifecycle updates.
• Give resumable shards one locked shard-owned dataset workspace; use attempt-local datasets when resume is disabled.
• Atomically replace attempt readiness snapshots with restrictive permissions and validated monotonic revisions.
• Validate semantic client results and candidate output manifests before winner publication.
• Publish exactly one immutable winner record per successful shard without overwrite.
• Reconcile persisted state with active and accounting scheduler observations in a fresh process.
• Classify retryable shards without winners and create new attempts from the pinned plan and artifact digests.
• Normalize and authorize collection destinations through the selected workspace and writable mount mapping.
• Submit collection as a zero-GPU CPU Slurm job using the run's pinned client image.
• Enumerate planned shards and validated winners only, process Parquet with bounded memory, stage beside the destination, verify row counts, and publish atomically without overwrite.

Acceptance criteria

• Failed, partial, stale, or scheduler-inconsistent attempts never publish winners.
• A winning shard is sealed against further retry.
• Concurrent resume writers, finalizers, retry requests, and collection requests fail or converge safely.
• Status reconciles attempt, readiness, generation, retry, and collection evidence without an in-memory controller.
• Accounting lag remains explicit and does not become guessed success or failure.
• Collection rejects missing, malformed, mismatched, or duplicate winners before reading bulk data.
• Collection never recursively globs attempt directories or selects a latest attempt.
• Large Parquet inputs are not processed by the login-host CLI.
• Interruption, validation failure, destination collision, timeout, and out-of-memory failure expose no partial final dataset.

Out of scope

• Generic object-store output.
• Cross-run resume or concurrent writers to one shard.
• Resident monitoring.
• Arbitrary unmanaged input trees for collection.

Investigation / Context

This is the persistent-state and collection lane for #850. Scheduler IDs are observations; stable run, shard, and attempt IDs remain the application identities.

Agent Plan / Findings

Implement persistence and locking first, then winner validation/publication, fresh-process reconciliation, retry, and winner-driven CPU collection.

Planned PR slices

1. Persistence and state-writer foundation

   • Status: Incomplete — PR not opened
   • Implement immutable run, shard, and attempt persistence; resolved-plan binding; restrictive permissions; locking; atomic publication; and monotonic readiness replacement.
   • Publish the state-writer API consumed by Implement Slurm submission and allocation runtime #868.
   • This slice provides the state foundation needed for M2.

2. Candidate and winner finalization

   • Status: Incomplete — PR not opened
   • Implement resumable-workspace ownership and locking, semantic client-result validation, candidate-output validation, and atomic no-overwrite winner publication.
   • Integrate client and candidate results from Implement the allocation-local Data Designer client worker #876.
   • This slice completes the M2 winner path.

3. Observation and reconciliation

   • Status: Incomplete — PR not opened
   • Reconcile persisted attempts, readiness, generation, and collection state from normalized active and accounting scheduler observations in a fresh process.
   • Preserve accounting lag as an explicit nonterminal state and apply the reviewed terminal-evidence precedence rules.
   • Consume normalized scheduler observations from 868#1.
   • This slice provides status and benchmark-refresh capabilities for M3d.

4. Retry and collection

   • Status: Incomplete — PR not opened
   • Classify retryable shards without winners and create new attempts from the pinned plan and artifact digests.
   • Submit winner-driven collection as a zero-GPU CPU job, validate planned winners, process data with bounded memory, and publish atomically without overwrite.
   • Consume submission primitives from 868#1 and the validated state produced by the preceding slices.
   • This slice supports M3c and completes the issue.

Each slice is an independently reviewable PR against feat/slurm-execution. Intermediate PRs reference Part of #869; only the final PR uses Closes #869.

Dependencies

The shared-contract and fake-infrastructure foundation tracked by #850 is complete. Slice 869#2 consumes client and candidate-result records from #876. Slices 869#3 and 869#4 consume normalized scheduler observations and submission primitives from 868#1. These scoped integrations do not block starting 869#1.