swift train ↔ vLLM rollout multimodal alignment

Background

In on-policy RL training (GRPO / OPD / GKD), rollout and training must consume the same multimodal inputs (input_ids, pixel_values, input_features, etc.). Otherwise, logprobs are computed on mismatched prompts/tensors.

This table tracks swift train vs vLLM rollout alignment per model family. The goal is to verify parity for GRPO/OPD and to expand coverage as we add more models and modalities.

— = modality not supported.
✅ = aligned.
❌ = not aligned (tracked upstream).

Alignment results

Known upstream issues (vLLM)

• Qwen3-VL / Qwen3.5 video — vLLM drops the outer <|vision_start|>/<|vision_end|> wrapper during video prompt expansion, causing a 2-token mismatch vs HuggingFace: #46817
• Gemma4 video — vLLM uses a per-frame image path and incorrect video metadata/timestamps vs Transformers Gemma4VideoProcessor: #46988

Start with the alignment table and the linked vLLM issues #46817 and #46988. Compare swift train and vLLM multimodal inputs for the listed model families and modalities, then update the alignment results when parity has been verified or a remaining mismatch is clearly recorded.