[Bug] Qwen3.5-VL grounding RL silently wrong: bundled SGLang image drops H/W rows of [3,T] M-RoPE (upstream sgl-project/sglang#35345, fix #35744)

TL;DR (English)

slime's default image slimerl/sglang:v0.5.15.post1-cu129 ships a SGLang version whose fused CUDA kernel fused_qk_gemma_rmsnorm_rope_gate reads Qwen3.5-VL's [3, T] M-RoPE positions as 1D [T] — it keeps only the temporal row and silently drops the height/width rows. Vision tokens get a corrupted 2D rotary encoding, so grounding / bbox rollouts are systematically wrong (~90px offset) while text/OCR look fine. In grounding RL this collapses the rollout rewards and the run looks like it is "degrading" when the model is fine — the rollout engine is encoding vision tokens wrong.

Root cause + fix are already upstream: sgl-project/sglang#35345 (kernel-level root cause) and fix PR sgl-project/sglang#35744 (extends the kernel to real [3,T] M-RoPE, keeps the fused fast path). This issue is a heads-up + workaround for slime users until that lands and the bundled image is rebuilt.

现象

用 slime 跑 Qwen3.5-VL 的 grounding / GUI 定位 RL 时,rollout 出的 bbox 系统性偏移约 90px,奖励迅速塌方,评测里定位命中率大幅下降(我这边 ~89% → ~44%),看着像"RL 把模型训退化了"。但 SFT 权重本身没问题——同一权重用 HF transformers / vLLM 推理 grounding 正常,是 rollout 推理这一侧就错,RL 只是把错误奖励放大。文本 / OCR / 数学任务不受影响。

根因

slime 的 docker/Dockerfile 默认 ARG SGLANG_IMAGE_TAG=v0.5.15.post1-cu129 / FROM slimerl/sglang:${SGLANG_IMAGE_TAG}。该 SGLang 版本里 Qwen3.5 的 full-attention 层走融合内核 fused_qk_gemma_rmsnorm_rope_gate(models/qwen3_5.py,_is_cuda and attn_output_gate 时),而该内核按 1D 取位置:

pos = tl.load(positions_ptr + token).to(tl.int64)

Qwen3.5-VL 多模态前向喂进来的是 [3, T] 的 M-RoPE(temporal / height / width 三行,mrope_section=[11,11,10]、mrope_interleaved=True、partial_rotary_factor=0.25)。内核只读了第一行 temporal,H/W 两维被静默丢弃 → vision token 的 2D 旋转位置编码坏掉。受影响模型类含 dense Qwen3_5ForConditionalGeneration 与 MoE Qwen3_5MoeForConditionalGeneration。

定位证据

组件级 bisection vs HF:pixel_values cos≈1.0、ViT+merger embeds cos≈0.9995、M-RoPE position ids 完全一致、GDN/线性层逐层无差异;第一个 full_attention 层 hidden states 发散约 10×。grounding center-in-box 命中率:融合(错)~26% → 修复后 ~81.7%(与 HF/vLLM 对齐)。

上游状态

• 根因 issue:sgl-project/sglang#35345
• 修复 PR:sgl-project/sglang#35744(等 run-ci 标签触发 CI)
• 另一组独立复现:sgl-project/sglang#35772

临时 workaround(上游合并 + 镜像重建前)

在容器内 patch python/sglang/srt/models/qwen3_5.py 的 dispatch,让 CUDA + M-RoPE 走 forward_prepare_fused_gate(正确的 Python MRotaryEmbedding);或在 slime 的 Dockerfile 里临时叠一层该 patch。我有个幂等的 apply/rollback 脚本可以分享。

建议:#35744 合并后,把 slime 默认的 SGLANG_IMAGE_TAG bump 到含修复的版本 / 重建 slimerl/sglang 镜像。

详细排查记录

slime强化训练之采坑记续续续-sglang 定位能力退化 — 知乎 https://zhuanlan.zhihu.com/p/2076401110785471102

Start with docker/Dockerfile and its SGLANG_IMAGE_TAG, then check upstream SGLang issues #35345 and PR #35744 for the fixed version. Update the bundled image reference after the upstream fix lands, rebuild the image, and verify Qwen3.5-VL grounding rollouts no longer show the reported bbox offset or reward collapse.