Delta weight sync (NCCL) produces NaN weights on Qwen3.5-122B MoE → "probability tensor contains inf/nan" crash

Summary

With --update-weight-mode delta --update-weight-transport nccl, the first delta apply on Qwen3.5-122B-A10B (grouped MoE) leaves NaN/inf in the rollout engine's weights. The next rollout crashes in sampling. Full-sync mode (--update-weight-mode full) on the same model/config is stable. Reproduced 4/4 runs.

Symptom

TensorCompare.cu:109 _assert_async_cuda_kernel: Assertion `probability tensor contains either inf, nan or element < 0` failed
Subprocess scheduler_0 crashed with exit code -3. Triggering SIGQUIT for cleanup...

Timeline: initial full seed sync ✓ → step-1 train ✓ → step-2 delta sync applied (checksum passes) ✓ → next rollout crashes.

Config

• slime with delta sync (#1806); SGLang v0.5.13 + docker/patch/latest/sglang.patch
• Qwen3.5-122B-A10B: --num-experts 256 --moe-grouped-gemm --moe-token-dispatcher-type alltoall --attention-backend flash (not flashinfer_trtllm)
• --update-weight-encoding indices, per-sync density ~1%
• Reproduced on both TP2/PP2/EP16 and TP2/PP4/EP8 (parallelism-independent)

Notes

• Checksum passes, so the (positions, values) payload arrives intact — corruption is in the apply into grouped-MoE params, not on the wire.
• Possibly related: #2193 (grouped-MoE GLU rechunk expert-axis), #2104 (failed apply not surfaced) — but neither reports this crash signature.

Start with the delta-sync apply path selected by --update-weight-mode delta and --update-weight-transport nccl, comparing it with the stable full-sync path. Reproduce on the Qwen3.5-122B-A10B grouped-MoE configuration and inspect the grouped-MoE parameter update after the checksum passes. Done means the first delta apply leaves no inf or NaN weights and the next rollout completes without the probability-tensor crash.