[cosmos3] Bitwise parity across LoRA updates: unmerged weight sync + engine-side adapter replay

[zhihengy]

Stacked on #109. Extends bitwise train/rollout parity from "before the first
optimizer step" to "after every weight update".

Problem

With the default merged LoRA sync, the rollout engine runs GEMM(W + sBA)
(one GEMM) while the trainer's peft forward runs
base(x) + lora_B(lora_A(x))*s (three GEMMs). The two are mathematically
equal but round differently in bf16, so model_output_mean_abs_diff jumps
from 0.0 to ~0.04 at step 2 (per-layer rounding gap amplified by 36 layers +
CFG) and stays there. A standalone probe confirms the mismatch on all cosmos3
LoRA shapes and confirms base+adapter replay is bitwise-exact.

Fix

Trainer — --lora-unmerged-weight-sync (opt-in, generic):
DiffusionUpdateWeightFromTensorLoRA ships base weights untouched, plus per
layer <prefix>.lora_A.weight / <prefix>.lora_B.weight (pre-rounded to the
forward dtype — the same bits FSDP's mixed-precision gather feeds the train
forward) and <prefix>.lora_scaling (fp64, so the engine recovers the exact
python float peft multiplies by).

Engine — cosmos3_bitwise patch group:

• Intercept adapter tensors at the weight-sync loader before sgl-d's
  warn-and-drop name filtering; parts are buffered across calls because the
  sender flushes one flattened bucket per dtype.
• Resolve targets through the model's own param-name mapping:
  add_q/k/v_proj -> to_qkv slices 0/1/2 (composes with the unfused-GEMM
  patch from [cosmos3] Bitwise train/rollout parity: FSDP precision spec + cosmos3_bitwise rollout patch group #109), to_add_out -> to_out (instance-level wrap).
• Replay peft 0.18's vanilla op sequence after each base GEMM:
  out + F.linear(F.linear(x, A), B) * s.
• Tensors are cloned out of the CUDA-IPC bucket storage the sender reclaims.

Verification

• GEMM probe (probe_lora_adapter_bitwise.py): adapter replay bitwise-equal
  to the peft path on all four cosmos3 LoRA shapes, with and without fused
  weight slicing; merged path differs at mean|diff| ~1.4e-1.
• Offline integration smoke: real cosmos3 name mapping + patched loader,
  parts arriving across two dtype buckets; attach points, slice ids, base
  loading, and forward bits all verified.
• 130-step Cosmos3-Nano GRPO run (PickScore, CFG 4.0, debug mode):
  train/model_output_mean_abs_diff = 0.0 and max_abs_diff = 0.0 on every
  step (previously 0.0 only at step 1). 144 adapters re-attached on each of
  the syncs; log-prob ratio deviation stays at the 1e-5 sampler-path floor.

[zhihengy]

Superseded by the rework now in #109 (commit 6d60e3d), which follows the #108 (qwen_image) patch style instead of a bespoke transport:

• The trainer ships adapters with the existing --lora-ipc-weight-sync flag (native LoRA-IPC path, weight_update_mode=lora_merge) — no new CLI flag, no trainer-side changes, and syncs only lora_A/lora_B instead of re-shipping all base weights.
• The engine side patches sglang's native LoRA wrappers in the cosmos3_bitwise group: set_lora_weights never merges and rounds A/B to the base weight dtype; wrapper forwards run eager base(x) + lora_B(lora_A(x))·s in peft's exact op order.
• Fused targets (add_q/k/v -> to_qkv) are routed to their merge slot via a patched _resolve_lora_ipc_layer_dict_key, replacing this PR's loader interception + _miles_lora registry.

Engine prerequisite (Cosmos3Pipeline + LoRAPipeline mixin) landed on sgl-project/sglang#34197.

Verified on Cosmos3-Nano GRPO: all 144 LoRA layer prefixes resolve (unmapped=0) and train/model_output_{mean,max}_abs_diff stay 0.0 across LoRA weight updates.