support checkpoint_no_save_replay_buffer

[hhaAndroid]

No description provided.

[hhaAndroid]

@claude review

[claude]

Nit: Backward compatibility — old checkpoints (saved before this PR) won't have the replay_buffer_saved key. The get(..., True) default handles this correctly, so existing checkpoints will always attempt replay buffer restore. Good.

[claude]

Claude: Code review completed for PR #1899 (support checkpoint_no_save_replay_buffer).

Summary

This PR adds a checkpoint_no_save_replay_buffer option that skips persisting the replay buffer during checkpointing. When enabled, the checkpoint saves a modified progress state that rolls back producer/target tracking to the last consumed position, so on resume the producer starts fresh with an empty buffer. The PR also adds two example RL configs and fixes CI test mocks for _save_eval_trajectories and _release_trace_store.

Issues

Warning

• agent_loop_manager.py:943-957: The progress state rollback logic in _progress_state_without_replay_buffer sets target_samples = consumed_samples, which means the deficit is 0 at resume. This is likely correct (the trainer's continue_produce() will advance targets), but the resume invariant is non-obvious and worth documenting with a brief comment.

Nit

• agent_loop_manager.py:1009: Backward compat with old checkpoints is handled correctly via get("replay_buffer_saved", True).
• test_agent_loop_manager_checkpoint.py:96-100: Test verifies buffer emptiness and model_step but could also assert on the restored next_consumer_step and producer_future_step to confirm progress state rollback.

ProduceBatchResult impact

produced_samples, produced_tokens, raw_rewards_sum, raw_rewards_count are zeroed in the saved state when no_save_replay_buffer=True. On resume, the first training step will re-produce from scratch. No impact on batch status or timing fields beyond what the empty-buffer restart implies.

RoutedExperts impact

Not affected — no changes to routed-experts ownership, object-ref plumbing, or rollout response handling.

Verdict

APPROVE — the core logic is sound, the backward-compat default is correct, and the test covers the main path. The one suggestion is to add a comment explaining the resume invariant.

[jayhenry]

ok, rollback producer_future_step and related states to next_consumer_step if drop replay_buffer