[ROADMAP][2026 Q3] Megatron Core Roadmap

This roadmap outlines the key Megatron Core features, enhancements, and improvements planned for Q3 2026. The 26.08 release has been delivered; remaining items track ongoing Q3 work. This is a tentative roadmap and subject to change.

**Last updated: September 15, 2026.**

For detailed information on past releases, see the [Megatron Core release notes](https://github.com/NVIDIA/Megatron-LM/releases). For the previous quarterly roadmap, see [Q2 2026 Roadmap #4997](https://github.com/NVIDIA/Megatron-LM/issues/4997). For component-specific plans and status, see the [Q3 MoE Roadmap #6757](https://github.com/NVIDIA/Megatron-LM/issues/6757) and [Q3 Megatron FSDP v2 Roadmap #6812](https://github.com/NVIDIA/Megatron-LM/issues/6812).

---

## Q3 Roadmap (26.08 delivered; ongoing Q3 work)

### MoE

- **DeepSeek-V4 architecture support** - Integrate CSA/HCA hybrid attention, DSA, and mHC; support Hash MoE and ClampedSwiGLU; add MTP with mHC, fused kernels, Muon/AdamW training recipes, packed sequence handling, and long-context validation ([tracker #4468](https://github.com/NVIDIA/Megatron-LM/issues/4468), [native hybrid orchestration #6402](https://github.com/NVIDIA/Megatron-LM/pull/6402), [Hash MoE #6403](https://github.com/NVIDIA/Megatron-LM/pull/6403), [fused mHC #6401](https://github.com/NVIDIA/Megatron-LM/pull/6401); foundational `dev` PRs [#4458](https://github.com/NVIDIA/Megatron-LM/pull/4458), [#4481](https://github.com/NVIDIA/Megatron-LM/pull/4481), [#4518](https://github.com/NVIDIA/Megatron-LM/pull/4518), and the load-balanced context-parallel DSA indexer [#6058](https://github.com/NVIDIA/Megatron-LM/pull/6058)).
- **Hash MoE and ClampedSwiGLU** - Implement Megatron Core support for DeepSeek-V4-specific routing and activation functions ([#4481](https://github.com/NVIDIA/Megatron-LM/pull/4481), [#5130](https://github.com/NVIDIA/Megatron-LM/pull/5130), [#6403](https://github.com/NVIDIA/Megatron-LM/pull/6403)).
- **MoE communication overlap** - ShortcutMoE and THD dispatcher support are delivered ([#6959](https://github.com/NVIDIA/Megatron-LM/pull/6959), [#5008](https://github.com/NVIDIA/Megatron-LM/pull/5008)). ShortcutMoE overlaps MoE all-to-all communication with paired attention, Mamba, or GDN computation; its PR reports 4.7–12.1% higher TFLOP/s and 82–95% overlap efficiency across four evaluated configurations. Latent shared-expert overlap and DeepEP V2 ElasticBuffer work continue ([#5135](https://github.com/NVIDIA/Megatron-LM/pull/5135), [#4632](https://github.com/NVIDIA/Megatron-LM/pull/4632)).
- **Optimized MoE kernels** - Optimize Blackwell performance by integrating GEMM + SwiGLU fused MLP, mHC, and batchedGEMM fusion techniques.
- **MegaMoE / ultra-sparse MoE** - Research expanded expert counts, sparsity-driven activations, predictive routing mechanisms, and adaptable recomputation strategies ([Megatron-LM #6810](https://github.com/NVIDIA/Megatron-LM/pull/6810), [Transformer Engine #3350](https://github.com/NVIDIA/TransformerEngine/pull/3350)).
- **MOK implementation** - Integrate Mixture-of-Kittens as a pluggable MoE megakernel backend ([#6572](https://github.com/NVIDIA/Megatron-LM/pull/6572)).

### Model Architecture and Support

- **DeepSeek-V4 recipes** - Complete recipes with end-to-end convergence validation across Megatron-LM and Megatron Bridge ([Megatron Bridge DeepSeek-V4 examples](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/deepseek_v4)).
- **Long-context validation** - Verify convergence for 4K/16K dense and 64K/1M sparse training setups.
- **HybridModel** - Introduce a heterogeneous-layer model definition for DeepSeek-V4 and future hybrid configurations, replacing GPTModel ([#4537](https://github.com/NVIDIA/Megatron-LM/pull/4537), [#5030](https://github.com/NVIDIA/Megatron-LM/pull/5030), [#5031](https://github.com/NVIDIA/Megatron-LM/pull/5031), [#4538](https://github.com/NVIDIA/Megatron-LM/pull/4538), [#4539](https://github.com/NVIDIA/Megatron-LM/pull/4539)).
- **Gated Delta Product (GDP) — delivered** - Add GDP as a first-class HybridModel mixer with training, static and dynamic inference, tensor and context parallelism, packed-sequence support, distributed checkpoint resharding, and optimizer integration ([#6074](https://github.com/NVIDIA/Megatron-LM/pull/6074)).
- **\* Ling-V3 Tiny support — delivered on `dev`** - Add training support for the KDA-MLA hybrid sparse-MoE architecture, including packed THD inputs, context parallelism, gated MLA, MTP, and checkpoint-resume validation ([#6556](https://github.com/NVIDIA/Megatron-LM/pull/6556)).
- **Nemotron 3 on HybridModel** - Make Nemotron 3 native to HybridModel.
- **\* DeepSeek-V4 transition to HybridModel** - Move DeepSeek-V4 development from GPTModel to the new HybridModel ([#5042](https://github.com/NVIDIA/Megatron-LM/pull/5042)).
- **Qwen3-Next support — delivered** - Support the GDN-based hybrid architecture, attention and shared-expert output gates, QK LayerNorm weight-decay handling, context parallelism, and packed sequences ([#1989](https://github.com/NVIDIA/Megatron-LM/pull/1989), [#2751](https://github.com/NVIDIA/Megatron-LM/pull/2751), [#2752](https://github.com/NVIDIA/Megatron-LM/pull/2752), [#2753](https://github.com/NVIDIA/Megatron-LM/pull/2753), [#2642](https://github.com/NVIDIA/Megatron-LM/pull/2642), [#2645](https://github.com/NVIDIA/Megatron-LM/pull/2645)).
- **GDN architecture and parallelism — delivered** - Support Gated Delta Net for Qwen3-Next, including context parallelism, packed sequences, and unified per-sequence AlltoAll ([#1989](https://github.com/NVIDIA/Megatron-LM/pull/1989), [#2642](https://github.com/NVIDIA/Megatron-LM/pull/2642), [#2645](https://github.com/NVIDIA/Megatron-LM/pull/2645), [#4913](https://github.com/NVIDIA/Megatron-LM/pull/4913)).
- **Fused GDN attention — WIP** - Transformer Engine fused GDN support is available ([Transformer Engine #3351](https://github.com/NVIDIA/TransformerEngine/pull/3351)); Megatron Core integration is being reapplied in draft [#7327](https://github.com/NVIDIA/Megatron-LM/pull/7327), following the revert of [#6645](https://github.com/NVIDIA/Megatron-LM/pull/6645).
- **GDN2 — in progress** - GDN2 architecture support is available in Megatron Core through Flash Linear Attention ([#5765](https://github.com/NVIDIA/Megatron-LM/pull/5765)). Optimized GDN2 kernels are available in [cuDNN Frontend](https://github.com/NVIDIA/cudnn-frontend/blob/main/python/cudnn/linear_attention/ops/gdn2.py), and Megatron Core integration is planned soon.
- **Qwen3.5 / Qwen3.5-VL support — in progress** - Add model architecture logic, packed sequence handling, Qwen3.5-VL validation, and bidirectional checkpoint conversion between Hugging Face and Megatron Core. Foundational `dev` work has landed ([#4748](https://github.com/NVIDIA/Megatron-LM/pull/4748), [#4749](https://github.com/NVIDIA/Megatron-LM/pull/4749), [#4750](https://github.com/NVIDIA/Megatron-LM/pull/4750), [#4751](https://github.com/NVIDIA/Megatron-LM/pull/4751)); HybridModel migration and pipeline-parallel integration continue ([#6315](https://github.com/NVIDIA/Megatron-LM/pull/6315), [#6081](https://github.com/NVIDIA/Megatron-LM/pull/6081)).
- **SFTDataset class** - Add a new pre-tokenized, packed implementation ([#5017](https://github.com/NVIDIA/Megatron-LM/pull/5017)).

### Parallelism

- **Megatron FSDP** - Continue maturing the FSDP2-compatible API, communication-compute overlap, activation recompute support, and pooled memory allocation. Delivered MFSDP v2 milestones include full-iteration CUDA Graphs, context parallelism, and independently configurable dense/expert hybrid sharding ([#7075](https://github.com/NVIDIA/Megatron-LM/pull/7075), [#7220](https://github.com/NVIDIA/Megatron-LM/pull/7220), [#7012](https://github.com/NVIDIA/Megatron-LM/pull/7012)).
- **Generalized Tensor Parallelism (GTP) — delivered** - Add fine-grained per-weight sharding and on-demand asynchronous all-gather with communication overlap; support CUDA Graphs, low-precision parameter gathers, composition with TP/SP/EP/DDP, Adam and Muon, and distributed checkpointing ([#4967](https://github.com/NVIDIA/Megatron-LM/pull/4967)).
- **NCCL EP support — delivered** - Add an expert-parallel transport path and dispatch-manager support ([#5129](https://github.com/NVIDIA/Megatron-LM/pull/5129), [#4644](https://github.com/NVIDIA/Megatron-LM/issues/4644)).
- **Next-generation dynamic DP** - Support multimodal data and improve performance.
- **Communication enhancements / DeepEPv2** - Integrate the DeepEPv2 token dispatcher and enable NCCL EP support.

### Performance and Memory

- **Deterministic training** - Track fixed-environment repeatability, baseline equivalence, restart replay, scale validation, and performance overhead ([#5785](https://github.com/NVIDIA/Megatron-LM/issues/5785)).
- **HybridModel 1F1B overlap — in progress** - The common schedule-plan foundation is delivered ([#4941](https://github.com/NVIDIA/Megatron-LM/pull/4941)); grouped HybridStack support, EP-overlap scheduling, FSDP, and training integration continue ([#4942](https://github.com/NVIDIA/Megatron-LM/pull/4942), [#4943](https://github.com/NVIDIA/Megatron-LM/pull/4943), [#4944](https://github.com/NVIDIA/Megatron-LM/pull/4944)).
- **Checkpoint load performance** - Reduce cross-rank reads for `torch_dist` load ([#4628](https://github.com/NVIDIA/Megatron-LM/pull/4628)).
- **Reduce MoE peak memory** - Avoid full intermediate list/cat buffers and chunk the return all-to-all/unpermute path.
- **Full-model CUDA Graph with paged stashing** - Advance paged stashing to enable full-iteration CUDA Graph capture and optimize memory for dropless MoE configurations.
- **Enhanced attention mechanisms** - Advance kernel development for GDN, NSA, and associated Transformer Engine or cuDNN Frontend integrations.
- **GDN memory optimization** - Implement fine-grained activation offloading and selective recompute strategies for in-projection, conv1d, and gated delta rule logic.

### Precision and Optimizers

- **MXFP8 and NVFP4 training paths** - DeepSeek-V3 NVFP4 training and FP4 execution in the 1F1B expert-parallel all-to-all overlap schedule are delivered ([#6841](https://github.com/NVIDIA/Megatron-LM/pull/6841), [#6135](https://github.com/NVIDIA/Megatron-LM/pull/6135)); continue improving MXFP8/NVFP4 parameter gathering and low-precision GroupedGEMM support.
- **Low-precision guide** - Document practical low-precision training recipes and optimizer configuration recipes.
- **Advanced Muon optimizer features** - Implement Newton-Schulz orthogonalization coefficients, FP8 primary weights, and precision-aware behavior for DeepSeek-V4.
- **MLA support for Muon** - Extend the existing metadata-tagging scheme for splitting MLA up-projections ([#5015](https://github.com/NVIDIA/Megatron-LM/pull/5015)).
- **Model-Optimizer PTQ and QAD** - Surface Model-Optimizer-based FP8/NVFP4 post-training quantization (PTQ) and quantization-aware distillation (QAD).
- **PTQ and QAD documentation** - Surface PTQ and QAD documentation using Nemotron 3 as examples.

### Inference

- **Sliding-window attention in dynamic batching** - Bring dynamic inference to parity with static Transformer Engine attention for SWA models ([#5138](https://github.com/NVIDIA/Megatron-LM/pull/5138)).
- **ETP** - Add support for expert tensor parallelism ([#4743](https://github.com/NVIDIA/Megatron-LM/pull/4743)).
- **Async scheduling — delivered** - Introduce asynchronous dynamic-inference scheduling and overlap, then enable it by default ([#5453](https://github.com/NVIDIA/Megatron-LM/pull/5453), [#5549](https://github.com/NVIDIA/Megatron-LM/pull/5549), [#5939](https://github.com/NVIDIA/Megatron-LM/pull/5939), [#6851](https://github.com/NVIDIA/Megatron-LM/pull/6851)).
- **Batch-invariant MoE inference — delivered** - Add fused vLLM and Transformer Engine backends for gated MoE plus routed block-scale MXFP8 FlashInfer execution with graph-safe bounded decode capacity ([#6521](https://github.com/NVIDIA/Megatron-LM/pull/6521), [#6864](https://github.com/NVIDIA/Megatron-LM/pull/6864)).

### Reinforcement Learning

- **Durable rollout storage — delivered** - Persist completed rollout groups in a checksummed write-through bank with recovery, compaction, and resume integration so completed work survives process interruptions ([#6352](https://github.com/NVIDIA/Megatron-LM/pull/6352)).
- **ReFIT and mutable low-precision weights — delivered** - Add NCCL many-to-many model transfer, in-place MXFP8 weight updates that preserve CUDA Graph buffer addresses, and parameter-aligned bounded-memory ReFIT execution ([#6712](https://github.com/NVIDIA/Megatron-LM/pull/6712), [#6826](https://github.com/NVIDIA/Megatron-LM/pull/6826), [#6878](https://github.com/NVIDIA/Megatron-LM/pull/6878)).

### Multimodal

- **MIMO (Multimodal In, Multimodal Out) extensions** - Expand the TP/DP heterogeneous primitive and enable autonomous nD parallelism for submodules.
- **Pipeline parallel support for the language model** - Allow colocated layouts where the destination language grid has PP > 1 while the source encoder grid remains PP = 1 ([#4784](https://github.com/NVIDIA/Megatron-LM/pull/4784)).
- **VLM CUDA Graph support** - Capture the language model submodule ([#4519](https://github.com/NVIDIA/Megatron-LM/pull/4519)).

\* : Feature is being developed on the `dev` branch first and may not be part of the 26.08 `main` release.

---

## How to Provide Feedback

We welcome community input on prioritization. Please:

1. React to items you would like prioritized.
2. Comment on this issue with use cases, constraints, and hardware / model configurations.
3. Open focused feature requests with the `enhancement` label.
4. Contribute pull requests for roadmap items where possible.

---

## Credits

This roadmap reflects the collective efforts of NVIDIA, external contributors, and the Megatron Core community.

Start by reviewing the Q3 roadmap sections and their linked issue and pull-request trackers, especially the MoE, HybridModel, parallelism, performance, precision, and inference items. This issue covers many ongoing and delivered efforts rather than naming one file or test; done would require maintaining accurate status and scope across the roadmap.

Written by the indexing model from the issue text.