NVIDIA / NVIDIA/Megatron-LM

Feature Request: Native Interleaved MRoPE Support for Qwen3.5

Open
#4,853 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

## Summary

Request to add native Megatron-LM support for Qwen3.5-style interleaved Multimodal RoPE (MRoPE). This support currently exists in Megatron Bridge, but not directly in native Megatron Core.

## Motivation

Users are interested in running Qwen3.5 fully through Megatron Core. Native Megatron Core already has MRoPE support, but it does not currently include the interleaved MRoPE layout used by Qwen3.5.

## Requested Support

1. Interleaved MRoPE Layout - Add Qwen3.5-style interleaved MRoPE support to the native `MultimodalRotaryEmbedding` / `mrope` path.
2. Config Wiring - Expose this as an opt-in config/argument, e.g. `mrope_interleaved`
3. Validation - Add parity tests against Hugging Face / Megatron-Bridge behavior for Qwen3.5 position embedding outputs.
4. Qwen3.5 Native Path - Enable this as part of native Megatron-LM Qwen3.5 support, for customers who require Megatron-LM directly rather than going through Megatron Bridge.

## References

* Hugging Face Qwen3.5 interleaved MRoPE implementation: https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen3_5/modeling_qwen3_5.py#L157
* Megatron-Bridge Qwen3-VL MRoPE implementation: https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/src/megatron/bridge/models/qwen_vl/modelling_qwen3_vl/rope.py

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.