Feature Request: Native Interleaved MRoPE Support for Qwen3.5
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
## Summary
Request to add native Megatron-LM support for Qwen3.5-style interleaved Multimodal RoPE (MRoPE). This support currently exists in Megatron Bridge, but not directly in native Megatron Core.
## Motivation
Users are interested in running Qwen3.5 fully through Megatron Core. Native Megatron Core already has MRoPE support, but it does not currently include the interleaved MRoPE layout used by Qwen3.5.
## Requested Support
1. Interleaved MRoPE Layout - Add Qwen3.5-style interleaved MRoPE support to the native `MultimodalRotaryEmbedding` / `mrope` path.
2. Config Wiring - Expose this as an opt-in config/argument, e.g. `mrope_interleaved`
3. Validation - Add parity tests against Hugging Face / Megatron-Bridge behavior for Qwen3.5 position embedding outputs.
4. Qwen3.5 Native Path - Enable this as part of native Megatron-LM Qwen3.5 support, for customers who require Megatron-LM directly rather than going through Megatron Bridge.
## References
* Hugging Face Qwen3.5 interleaved MRoPE implementation: https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen3_5/modeling_qwen3_5.py#L157
* Megatron-Bridge Qwen3-VL MRoPE implementation: https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/src/megatron/bridge/models/qwen_vl/modelling_qwen3_vl/rope.py
Contributor guide
Assessment
This issue has not been assessed yet.