Materialization claim needs an identity, not just CLEAN (#92)

[jaredLunde]

Closes #92.

The bug is real

write_materialized gated its commit on state == CLEAN. CLEAN is not an identity. Across a slow S3 fetch the claimed block can cycle:

CLEAN(M) --steal--> DIRTY --flush--> SYNCING --upload+evict--> NOT_PRESENT --re-claim--> CLEAN(other)

The check passes on that foreign claim, and M pwrites its fetched pre-image over data that was already acknowledged and uploaded to S3. Silent rollback of a durable write.

Where the re-claim comes from

Not from try_claim_block — that linearizes through the promote claim and cannot run while a materialization is live. It comes from cache.pre_write → CacheInner::set_present: the ublk ZC path marks its blocks present and only lands the guest bytes afterwards, via the kernel WRITE_FIXED, so it cannot hold the claim across the transition.

The per-block lock_write_range does not cover this either — it is taken by BlockHandler::write only, not by pre_write/zc_prepare_write, trim, or write_zeroes. So the "already serialized by the write lock" reasoning holds for USER_COPY and not for the ZC path.

The fix: give every claim a ClaimToken

• PromoteClaimBitmap becomes a sparse claim registry (HashMap<usize, ClaimSlot> — same one-entry-per-in-flight-claim cost as the HashSet it replaces) carrying a monotonic token plus a valid bit.
• CacheInner::set_present — the one NOT_PRESENT → CLEAN that bypasses the claim — invalidates the live claim before flipping the state, so a holder that reads CLEAN and then finds its token valid is guaranteed to be looking at its own claim.
• write_materialized takes the MaterializationClaim instead of a bare block index, and checks state + token in one critical section (if_valid holds the claim map across the pwrite; the data_file WRITE lock is already held there, which excludes everything else the map guards).
• MaterializationClaim::unclaim is claim-checked too: an aborting materializer must not knock a later writer's CLEAN back to NOT_PRESENT and strand its in-flight data write.

Cost

Steady state is unchanged: set_present early-returns on is_present before touching the claim map, so an overwrite of an already-present block never takes the mutex. Only a genuine NOT_PRESENT → CLEAN pays a lock — on a block that is about to do an S3 GET.

cargo bench --bench workloads (random 4/16/64 KB + sequential 1/10 MB), with vs. without the change: every delta inside run-to-run noise (p > 0.05 on 4 of 5; the one "significant" result had the without-fix run slower).

Verification

• New regression test write_materialized_aborts_when_block_was_reclaimed walks the full cycle (claim → steal → flush → evict → re-claim → guest bytes land) and asserts the stale pre-image does not land. It fails without the invalidation.
• 585 lib tests, 222 integration tests green.
• fio_verify_random_cold_wake through a real ublk device: CRC32C verification passed.
• stateright-model (17 tests) still green — it is a standalone crate. Note it cannot reach this bug: faithful.rs collapses the claim-first materialization into one atomic step reading the live s3[b] ("nothing can change it while CLEAN" — the exact assumption the ABA violates), and has no separate "ZC pre_write marks present, data lands later" transition. Extending the model is a follow-up.

🤖 Generated with Claude Code

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.