Megatron-LM Monthly Activity Report — July 2026 (Final, through July 31, 2026)

# Megatron-LM Monthly Activity Report — July 2026 (Final, through July 31, 2026)

> Chinese version: [中文版本](https://github.com/lhb8125/megatron-lm-activity-report/blob/main/reports/zh-CN/2026/07.md)

> Reporting window: 2026-07-01 through 2026-07-31 (Asia/Shanghai); source: [`NVIDIA/Megatron-LM`](https://github.com/NVIDIA/Megatron-LM).

## Overview

Delivered work centered on DeepSeek-V4 attention, scalable parallelism, MoE execution, checkpoint reliability, CUDA Graph safety, hybrid sequence models, dynamic inference, and multimodal training. Ongoing work extends these areas with MFSDP v2, variable-length THD execution, new attention mixers, multimodal pipelines, and RL/inference optimizations.

- Substantive PR activity after filtering: opened 436; merged 208; closed without merge 139; active at cutoff 220.
- Excluded from the report and statistics: test-only 40, CI-only 34, test/CI-only 0, format-only 0.

## Delivered

### DeepSeek-V4 CSA/DSA Context Parallelism and Fused THD Backends

DeepSeek-V4 attention gained context-parallel layouts, packed-sequence support, TileLang and cuDNN fused DSA backends, precision controls, HybridModel integration, and corrected indexer-gradient behavior. Ordinary DSA functionality was also migrated to dev while retaining its advanced compressed-attention integrations.

- Context-parallel training now covers DeepSeek-V4 CSA and DSA attention with packed sequences.
- TileLang and cuDNN implementations expose fused indexer and sparse-MLA execution through the standard DSA path.
- CSA components that require higher precision remain outside FP8 initialization or in FP32, and zero-coefficient indexer losses retain explicit gradients.
- HybridModel THD, MoE auxiliary-loss, and compressed-attention configuration behavior was aligned with the broader model stack.

Related PRs: [#5049](https://github.com/NVIDIA/Megatron-LM/pull/5049), [#5087](https://github.com/NVIDIA/Megatron-LM/pull/5087), [#5099](https://github.com/NVIDIA/Megatron-LM/pull/5099), [#5308](https://github.com/NVIDIA/Megatron-LM/pull/5308), [#5762](https://github.com/NVIDIA/Megatron-LM/pull/5762), [#5809](https://github.com/NVIDIA/Megatron-LM/pull/5809), [#6020](https://github.com/NVIDIA/Megatron-LM/pull/6020)

### Megatron-FSDP Overlap, HSDP, and Optimizer Lifecycle

Experimental Megatron-FSDP gained parameter-gather and gradient-reduction overlap, deferred HSDP outer-DP synchronization, symmetric-memory staging, reusable hybrid buffers, and lifecycle fixes spanning root hooks, microbatch state, fused grouped experts, optimizer synchronization, and CPU-offloaded FP32-retained parameters.

- Forward and backward prefetching, event-scoped waits, and separate execution orders overlap collectives with adjacent computation.
- HSDP accumulates inner-DP gradients across microbatches and performs the outer-DP reduction only on the final microbatch.
- Symmetric-memory staging, shared contexts, and minimal reusable bucket pools support more efficient collective and hybrid-model execution.
- Root and fused-expert hooks now materialize required parameters, while optimizer initialization, weight synchronization, and selective FP32 CPU offload preserve update correctness.

Related PRs: [#5440](https://github.com/NVIDIA/Megatron-LM/pull/5440), [#5462](https://github.com/NVIDIA/Megatron-LM/pull/5462), [#5513](https://github.com/NVIDIA/Megatron-LM/pull/5513), [#5636](https://github.com/NVIDIA/Megatron-LM/pull/5636), [#5652](https://github.com/NVIDIA/Megatron-LM/pull/5652), [#5719](https://github.com/NVIDIA/Megatron-LM/pull/5719), [#5743](https://github.com/NVIDIA/Megatron-LM/pull/5743), [#5808](https://github.com/NVIDIA/Megatron-LM/pull/5808), [#5949](https://github.com/NVIDIA/Megatron-LM/pull/5949), [#5976](https://github.com/NVIDIA/Megatron-LM/pull/5976), [#6124](https://github.com/NVIDIA/Megatron-LM/pull/6124)

### Generalized Tensor Parallelism and Low-Precision Weight Movement

Generalized Tensor Parallelism introduced per-weight sharding, asynchronous on-demand all-gather, gradient reduction, low-precision parameter gathering, CUDA Graph support, and optimizer integration. Follow-up work added grouped-expert prefetching and repaired full-iteration graph capture and noncontiguous tensor reductions.

- The new composable sharding path reduces persistent parameter memory and can overlap communication with computation.
- Grouped-expert weights can be prefetched one block ahead.
- Full-iteration graph capture now materializes gradient nodes on the capture stream and rejoins asynchronous collectives.
- Tensor-parallel reduction now returns the all-reduced result for noncontiguous inputs instead of silently returning the original tensor.

Related PRs: [#4967](https://github.com/NVIDIA/Megatron-LM/pull/4967), [#5338](https://github.com/NVIDIA/Megatron-LM/pull/5338), [#6057](https://github.com/NVIDIA/Megatron-LM/pull/6057), [#6077](https://github.com/NVIDIA/Megatron-LM/pull/6077)

### NCCL/DeepEP MoE Dispatch, Quantized Experts, and Routing Integrity

MoE execution added NCCL expert parallelism, zero-copy NCCL dispatch, THD support for DeepEP and HybridEP, inference-optimized Qwen experts, and grouped-GEMM shared experts. Reliability work addressed quantized grouped weights, TP/EP gradient norms, padding, auxiliary-loss scaling, token-count equalization, and fused-expert bias semantics.

- NCCL expert-parallel communication integrates with overlapped 1F1B schedules, while zero-copy dispatch avoids supported staging-buffer copies.
- THD-packed training can use DeepEP and HybridEP, and Qwen-style inference can select fused expert implementations.
- BF16, MXFP8, and NVFP4 grouped-expert handling and topology-aware gradient-norm accounting were corrected.
- Padding is excluded from training and inference routing; quantile balancing and bounded routing traces provide additional load-balancing and diagnostic options.

Related PRs: [#4922](https://github.com/NVIDIA/Megatron-LM/pull/4922), [#5008](https://github.com/NVIDIA/Megatron-LM/pull/5008), [#5129](https://github.com/NVIDIA/Megatron-LM/pull/5129), [#5220](https://github.com/NVIDIA/Megatron-LM/pull/5220), [#5349](https://github.com/NVIDIA/Megatron-LM/pull/5349), [#5464](https://github.com/NVIDIA/Megatron-LM/pull/5464), [#5487](https://github.com/NVIDIA/Megatron-LM/pull/5487), [#5542](https://github.com/NVIDIA/Megatron-LM/pull/5542), [#5604](https://github.com/NVIDIA/Megatron-LM/pull/5604), [#5668](https://github.com/NVIDIA/Megatron-LM/pull/5668), [#5700](https://github.com/NVIDIA/Megatron-LM/pull/5700), [#5735](https://github.com/NVIDIA/Megatron-LM/pull/5735), [#5798](https://github.com/NVIDIA/Megatron-LM/pull/5798), [#5850](https://github.com/NVIDIA/Megatron-LM/pull/5850), [#5916](https://github.com/NVIDIA/Megatron-LM/pull/5916), [#6099](https://github.com/NVIDIA/Megatron-LM/pull/6099)

### Streaming Quantized Checkpoints and Cross-Layout Optimizer State

Checkpoint and optimizer-state handling gained streamed low-precision dequantization, GPT-to-Hybrid retargeting, FSDP2 SwiGLU reconstruction, mixed-dtype and layer-wise optimizer repairs, replica-scoped validation, module-specific process groups, and lower checkpoint-path synchronization overhead.

- FP8, MXFP8, and FP4 loads now allocate BF16 scratch per tensor, reducing transient memory to approximately the largest tensor.
- GPT checkpoints can be loaded and resharded into HybridModel across supported distributed layouts without offline conversion.
- FSDP2 SwiGLU, mixed-dtype optimizer groups, layer-wise Torch checkpoints, chained optimizers, Lion state, and scheduler fields received targeted restoration fixes.
- Checkpoint validation can use a supplied process group, and timing no longer includes unnecessary barriers.

Related PRs: [#4451](https://github.com/NVIDIA/Megatron-LM/pull/4451), [#4705](https://github.com/NVIDIA/Megatron-LM/pull/4705), [#5388](https://github.com/NVIDIA/Megatron-LM/pull/5388), [#5470](https://github.com/NVIDIA/Megatron-LM/pull/5470), [#5557](https://github.com/NVIDIA/Megatron-LM/pull/5557), [#5579](https://github.com/NVIDIA/Megatron-LM/pull/5579), [#5684](https://github.com/NVIDIA/Megatron-LM/pull/5684), [#5714](https://github.com/NVIDIA/Megatron-LM/pull/5714), [#5742](https://github.com/NVIDIA/Megatron-LM/pull/5742), [#5759](https://github.com/NVIDIA/Megatron-LM/pull/5759), [#5792](https://github.com/NVIDIA/Megatron-LM/pull/5792), [#5835](https://github.com/NVIDIA/Megatron-LM/pull/5835), [#6048](https://github.com/NVIDIA/Megatron-LM/pull/6048), [#6073](https://github.com/NVIDIA/Megatron-LM/pull/6073), [#6092](https://github.com/NVIDIA/Megatron-LM/pull/6092)

### CUDA Graph Activation Lifetimes, Offloading, and Packed-THD Safety

CUDA Graph reliability improved across activation offloading, backward-required tensor lifetimes, partial-graph memory retention, inference buffers, context-parallel training, packed-THD metadata, sampling, and DDP initialization streams.

- Fine-grained activation offloading now uses synchronized streams, events, and deferred commits during Transformer Engine capture and replay.
- Observer-based lifetime marking prevents graph buffer reuse from overwriting tensors needed by backward.
- Partial graphs retain less memory, and inference all-gather storage is preallocated so recorded graphs never reference resized buffers.
- Context-parallel training and padded THD batches now follow consistent graph behavior, while graph-unsafe FlashInfer sampling is disabled.

Related PRs: [#3697](https://github.com/NVIDIA/Megatron-LM/pull/3697), [#5451](https://github.com/NVIDIA/Megatron-LM/pull/5451), [#5546](https://github.com/NVIDIA/Megatron-LM/pull/5546), [#5635](https://github.com/NVIDIA/Megatron-LM/pull/5635), [#5724](https://github.com/NVIDIA/Megatron-LM/pull/5724), [#5791](https://github.com/NVIDIA/Megatron-LM/pull/5791), [#5894](https://github.com/NVIDIA/Megatron-LM/pull/5894), [#5975](https://github.com/NVIDIA/Megatron-LM/pull/5975), [#6021](https://github.com/NVIDIA/Megatron-LM/pull/6021)

### Gated DeltaNet Chunkwise CP, Hybrid MLA, and Fused pre-GDR

Hybrid sequence-model execution expanded with chunkwise context parallelism for Gated DeltaNet, MLA and absorbed-MLA layers in HybridModel, fused pre-GDR for SBHD and packed THD layouts, and correct MLA prefill-versus-decode selection.

- Gated DeltaNet now uses a Transformer Engine-integrated chunkwise context-parallel layout.
- HybridModel supports MLA and absorbed MLA with QK-normalization configuration resolution.
- Fused pre-GDR covers forward and backward chunkwise CP, sequence boundaries, and headwise parameter slicing.
- The reported BF16 benchmark using 397B GDN dimensions measured 2.68x to 3.71x fused-versus-unfused speedups across CP sizes one through eight.

Related PRs: [#3282](https://github.com/NVIDIA/Megatron-LM/pull/3282), [#4452](https://github.com/NVIDIA/Megatron-LM/pull/4452), [#4902](https://github.com/NVIDIA/Megatron-LM/pull/4902), [#5638](https://github.com/NVIDIA/Megatron-LM/pull/5638)

### Dynamic Inference Scheduling, Prefix Caches, and KV/SSM Transfer

Dynamic inference gained asynchronous phase scheduling, disaggregated KV and SSM transfer through NCCL or NIXL, load-aware prefix routing, and extensive correctness work for bookkeeping buffers, dummy blocks, sampling, KV reference counts, and Mamba prefix states.

- A pending GPU forward can overlap CPU request processing, scheduling preparation, and sampled-token materialization.
- Disaggregated deployments can transfer KV, Mamba, and SSM state through a common NCCL or NIXL interface.
- Prefix-cache routing and eviction now account for worker load, tree structure, reference counts, and active-request ownership.
- Mamba prefix caching supports idle forwards, multi-chunk prompts, whole-prefix skipping, optimized state extraction, and substantially smaller scratch reservations.

Related PRs: [#5502](https://github.com/NVIDIA/Megatron-LM/pull/5502), [#5549](https://github.com/NVIDIA/Megatron-LM/pull/5549), [#5607](https://github.com/NVIDIA/Megatron-LM/pull/5607), [#5715](https://github.com/NVIDIA/Megatron-LM/pull/5715), [#5822](https://github.com/NVIDIA/Megatron-LM/pull/5822), [#5861](https://github.com/NVIDIA/Megatron-LM/pull/5861), [#5863](https://github.com/NVIDIA/Megatron-LM/pull/5863), [#5866](https://github.com/NVIDIA/Megatron-LM/pull/5866), [#5939](https://github.com/NVIDIA/Megatron-LM/pull/5939), [#5950](https://github.com/NVIDIA/Megatron-LM/pull/5950), [#5952](https://github.com/NVIDIA/Megatron-LM/pull/5952), [#5983](https://github.com/NVIDIA/Megatron-LM/pull/5983), [#6047](https://github.com/NVIDIA/Megatron-LM/pull/6047)

### Megatron Lite DeepSeek-V4 RL and Heterogeneous MIMO Training

Experimental post-training coverage expanded through Megatron Lite model and runtime synchronization, an end-to-end DeepSeek-V4 Flash RL stack, heterogeneous non-colocated MIMO training, Agent Compose runtime components, and MIMO optimizer, prefetch, and DDP-overlap fixes.

- Megatron Lite added Kimi K2, GLM5, and DeepSeek-V4 implementations plus checkpointing, export, FSDP2 restore, pipeline automation, and post-training workflows.
- DeepSeek-V4 Flash RL covers THD packing, attention and MTP fixes, rollout resynchronization, FSDP2 handling, and streaming checkpoint export.
- Heterogeneous MIMO can place components on separate process grids while keeping optimizer-success decisions consistent globally.
- Frozen-encoder prefetch and inner-DDP lifecycle hooks overlap preparation and communication without omitting required synchronization.

Related PRs: [#5331](https://github.com/NVIDIA/Megatron-LM/pull/5331), [#5577](https://github.com/NVIDIA/Megatron-LM/pull/5577), [#5602](https://github.com/NVIDIA/Megatron-LM/pull/5602), [#5694](https://github.com/NVIDIA/Megatron-LM/pull/5694), [#5744](https://github.com/NVIDIA/Megatron-LM/pull/5744), [#5833](https://github.com/NVIDIA/Megatron-LM/pull/5833), [#5862](https://github.com/NVIDIA/Megatron-LM/pull/5862), [#5937](https://github.com/NVIDIA/Megatron-LM/pull/5937), [#5979](https://github.com/NVIDIA/Megatron-LM/pull/5979)

### NeMo-Compatible Audio Encoder and Qwen3.5-VL Fused mRoPE

Multimodal foundations added a standalone NeMo-compatible audio encoder and waveform processor, together with fused multimodal RoPE kernels for Qwen2-VL and Qwen3.5-VL training.

- The audio stack includes log-mel preprocessing, dense and packed forwards, LM projection, and .nemo checkpoint loading without NeMo runtime dependencies.
- Waveform processing can materialize encoder features and estimate packed-data frame and embedding counts without Megatron-Energon types.
- Triton mRoPE forward and backward kernels support BSHD and THD layouts, context parallelism, and FP16, BF16, and FP32 inputs.

Related PRs: [#5565](https://github.com/NVIDIA/Megatron-LM/pull/5565), [#5570](https://github.com/NVIDIA/Megatron-LM/pull/5570), [#5962](https://github.com/NVIDIA/Megatron-LM/pull/5962)

## In Progress

### DeepSeek-V4 CSA/DSA Fusion, CP Load Balancing, and Sparse FLOPs

Ongoing DeepSeek-V4 work is reconstructing the HybridModel stack on main while improving compressed sparse attention communication, compact low-precision indexers, fused THD pooling, CP load balancing, sparse FLOPs accounting, offline-packed THD compatibility, and selective-recompute weight prefetch.

- The integration series covers compressed sparse attention, THD execution, mHC, MoE, MTP, and checkpoint prerequisites; GLM5.2 adds shared DSA indices.
- Indexer-K and compressed-KV communication can overlap independent attention work, while CP redistribution removes dense per-rank score buffers.
- Compact BF16/MXFP8 indexers, fused gated pooling, and padded-index skipping target lower kernel and memory costs.
- Correctness work addresses CSA teacher normalization, Triton Q-RoPE races, sparse FLOPs, gradient checkpoint selection, and offline-packed supervised fine-tuning.

Related PRs: [#5691](https://github.com/NVIDIA/Megatron-LM/pull/5691), [#5795](https://github.com/NVIDIA/Megatron-LM/pull/5795), [#5875](https://github.com/NVIDIA/Megatron-LM/pull/5875), [#5899](https://github.com/NVIDIA/Megatron-LM/pull/5899), [#5960](https://github.com/NVIDIA/Megatron-LM/pull/5960), [#5963](https://github.com/NVIDIA/Megatron-LM/pull/5963), [#5984](https://github.com/NVIDIA/Megatron-LM/pull/5984), [#5992](https://github.com/NVIDIA/Megatron-LM/pull/5992), [#6058](https://github.com/NVIDIA/Megatron-LM/pull/6058), [#6148](https://github.com/NVIDIA/Megatron-LM/pull/6148), [#6158](https://github.com/NVIDIA/Megatron-LM/pull/6158), [#6166](https://github.com/NVIDIA/Megatron-LM/pull/6166), [#6176](https://github.com/NVIDIA/Megatron-LM/pull/6176)

### MFSDP v2 Composition, ZeRO Layouts, Symmetric Memory, and Prefetch

MFSDP v2 is being integrated with MCore optimizers, tensor and expert parallelism, multiple ZeRO layouts, DP-outer sharding, symmetric-memory configuration, tied parameters, activation recomputation, and combined-1F1B prefetch. Experimental FSDP overlap work also explores zero-CTA and dual-communicator collectives.

- Adapter and optimizer paths introduce composable v2 construction while retaining explicit rejection of unsupported configurations.
- TP composition keeps compute parameters TP-local while using full-mesh DTensors for optimizer state and checkpoints; expert parallelism receives dedicated mesh and gradient scaling.
- No-shard, ZeRO-1, ZeRO-2, and ZeRO-3 layouts extend the available parameter, gradient, and optimizer-state placement choices.
- Zero-CTA and dual-communicator paths aim to preserve SM capacity and reduce collective blocking, although the reported zero-CTA throughput regression remains under investigation.

Related PRs: [#5657](https://github.com/NVIDIA/Megatron-LM/pull/5657), [#5772](https://github.com/NVIDIA/Megatron-LM/pull/5772), [#5774](https://github.com/NVIDIA/Megatron-LM/pull/5774), [#5813](https://github.com/NVIDIA/Megatron-LM/pull/5813), [#5836](https://github.com/NVIDIA/Megatron-LM/pull/5836), [#5865](https://github.com/NVIDIA/Megatron-LM/pull/5865), [#5954](https://github.com/NVIDIA/Megatron-LM/pull/5954), [#5956](https://github.com/NVIDIA/Megatron-LM/pull/5956), [#5972](https://github.com/NVIDIA/Megatron-LM/pull/5972), [#5985](https://github.com/NVIDIA/Megatron-LM/pull/5985), [#6052](https://github.com/NVIDIA/Megatron-LM/pull/6052), [#6127](https://github.com/NVIDIA/Megatron-LM/pull/6127), [#6137](https://github.com/NVIDIA/Megatron-LM/pull/6137), [#6153](https://github.com/NVIDIA/Megatron-LM/pull/6153), [#6164](https://github.com/NVIDIA/Megatron-LM/pull/6164), [#6169](https://github.com/NVIDIA/Megatron-LM/pull/6169), [#6174](https://github.com/NVIDIA/Megatron-LM/pull/6174)

### THD and Partial CUDA Graphs with Dynamic CP and MoE Paged Stash

CUDA Graph work is extending capture to dynamic context-parallel sizes, variable-length THD tensors, Megatron-FSDP, mHC recomputation, nested inputs, paged MoE stash, GTP communication overlap, local graph scopes, and MLA cache updates.

- Layer-wise graph banks select among multiple dynamic-CP sizes while preserving THD metadata, process groups, and MLA RoPE lifetimes.
- Partial and full-iteration graph paths cover packed THD training and planned FSDP storage.
- mHC recomputation and paged stash coordinate static-buffer lifetimes and capture-specific schedule ordering.
- Nested input reuse, dense-layer initialization, viewless pipeline outputs, and stream ownership address graph construction and replay correctness.

Related PRs: [#5618](https://github.com/NVIDIA/Megatron-LM/pull/5618), [#5783](https://github.com/NVIDIA/Megatron-LM/pull/5783), [#5800](https://github.com/NVIDIA/Megatron-LM/pull/5800), [#5807](https://github.com/NVIDIA/Megatron-LM/pull/5807), [#5825](https://github.com/NVIDIA/Megatron-LM/pull/5825), [#5841](https://github.com/NVIDIA/Megatron-LM/pull/5841), [#5931](https://github.com/NVIDIA/Megatron-LM/pull/5931), [#6022](https://github.com/NVIDIA/Megatron-LM/pull/6022), [#6060](https://github.com/NVIDIA/Megatron-LM/pull/6060), [#6121](https://github.com/NVIDIA/Megatron-LM/pull/6121), [#6167](https://github.com/NVIDIA/Megatron-LM/pull/6167), [#6168](https://github.com/NVIDIA/Megatron-LM/pull/6168)

### Sliding-Window, NVSHMEM, and Variable-Length THD Context Parallelism

Context-parallel execution is expanding through sliding-window halo exchange, Gated DeltaNet chunkwise layouts, automatic per-layer layout conversion, NVSHMEM attention, causal Conv1d, and an end-to-end variable-length THD data and scheduling stack.

- Sliding-window CP exchanges only required causal K/V halos and returns their gradients to owning ranks.
- Explicit layout metadata converts between zigzag or contiguous SBHD and THD representations at layer boundaries.
- Variable-length datasets, DP-balanced packing, packed-batch distribution, dynamic microbatch counts, and GPT training integration form an end-to-end THD workflow.
- FLOPs statistics are recorded only on virtual pipeline stage zero to avoid inflation by the virtual-pipeline size.

Related PRs: [#5597](https://github.com/NVIDIA/Megatron-LM/pull/5597), [#5637](https://github.com/NVIDIA/Megatron-LM/pull/5637), [#5664](https://github.com/NVIDIA/Megatron-LM/pull/5664), [#5904](https://github.com/NVIDIA/Megatron-LM/pull/5904), [#5905](https://github.com/NVIDIA/Megatron-LM/pull/5905), [#5907](https://github.com/NVIDIA/Megatron-LM/pull/5907), [#5909](https://github.com/NVIDIA/Megatron-LM/pull/5909), [#5910](https://github.com/NVIDIA/Megatron-LM/pull/5910), [#5917](https://github.com/NVIDIA/Megatron-LM/pull/5917), [#5969](https://github.com/NVIDIA/Megatron-LM/pull/5969), [#5993](https://github.com/NVIDIA/Megatron-LM/pull/5993), [#6075](https://github.com/NVIDIA/Megatron-LM/pull/6075)

### Qwen3.5-VL Pipeline Parallelism, Dynamic Batching, and Vision FLOPs

Multimodal work is adding dynamically batched vision-language requests, variable-length Qwen3.5-VL mock data, pipeline-stage construction, modality-aware MTP masking, regular regression coverage, and vision-encoder FLOPs accounting.

- Dynamic batching carries images, expanded placeholders, request-scoped embeddings, masks, and expert-parallel dummy inputs without changing text-only behavior.
- Qwen3.5-VL can split language and vision work across pipeline stages with stage-aware inputs and embedding sharing.
- MTP loss masks exclude modality embeddings from supervised targets.
- Vision FLOPs aggregate variable-resolution image-grid statistics while retaining decoder-only behavior and nominal-metadata fallback.

Related PRs: [#5606](https://github.com/NVIDIA/Megatron-LM/pull/5606), [#5927](https://github.com/NVIDIA/Megatron-LM/pull/5927), [#6010](https://github.com/NVIDIA/Megatron-LM/pull/6010), [#6081](https://github.com/NVIDIA/Megatron-LM/pull/6081), [#6083](https://github.com/NVIDIA/Megatron-LM/pull/6083), [#6173](https://github.com/NVIDIA/Megatron-LM/pull/6173)

### Kimi Delta Attention, GDN2, Mamba2 SSD, and Hybrid Mixer Recipes

Hybrid sequence-model development is adding Kimi Delta Attention, GDN2 and Gated Delta Product mixers, a CuTe DSL Mamba2 SSD backend, speculative Mamba state updates, selective GDN recomputation, deterministic Triton configuration, and a declarative hybrid-model recipe DSL.

- Kimi Delta Attention implements per-key-channel decay, short convolution, gated normalization, and TP/SP-aware state handling.
- GDN2 and Gated Delta Product extend HybridModel across distributed training, packed sequences, inference caches, checkpoint resharding, and optimizer integration.
- The CuTe DSL Mamba2 SSD backend reported approximately 5%–7% end-to-end prefill gains for the tested 2K–8K inputs.
- Deterministic kernel configuration pins and verifies Triton tuning choices across ranks, while shared GDN/GDN2 packaging reduces duplicated logic.

Related PRs: [#5765](https://github.com/NVIDIA/Megatron-LM/pull/5765), [#5769](https://github.com/NVIDIA/Megatron-LM/pull/5769), [#5803](https://github.com/NVIDIA/Megatron-LM/pull/5803), [#5829](https://github.com/NVIDIA/Megatron-LM/pull/5829), [#5982](https://github.com/NVIDIA/Megatron-LM/pull/5982), [#6066](https://github.com/NVIDIA/Megatron-LM/pull/6066), [#6074](https://github.com/NVIDIA/Megatron-LM/pull/6074), [#6076](https://github.com/NVIDIA/Megatron-LM/pull/6076), [#6088](https://github.com/NVIDIA/Megatron-LM/pull/6088)

### Asynchronous Checkpoint Loading, FSDP Conversion, and Safe Deserialization

Checkpoint work is adding asynchronous file loading, CPU-only FSDP format conversion, MFSDP v2 DCP support, safer deserialization, and fixes for Mixtral, tied embeddings, hybrid models, optimizer padding, and checkpoint allocator behavior.

- Reusable load plans and pinned CPU tensors allow disk reads and memory copies to overlap other work without background-thread collectives.
- A model-free converter transforms fsdp_dtensor model and optimizer state into native torch_dist checkpoints on CPU.
- MFSDP v2 DCP metadata handles uneven DTensor shards and resynchronizes compute weights after loading.
- Safety and reliability work restricts an affected load path to weights, repairs Mixtral conversion, and validates whether restart changes truly improve first-iteration recovery.

Related PRs: [#5578](https://github.com/NVIDIA/Megatron-LM/pull/5578), [#5671](https://github.com/NVIDIA/Megatron-LM/pull/5671), [#5678](https://github.com/NVIDIA/Megatron-LM/pull/5678), [#5737](https://github.com/NVIDIA/Megatron-LM/pull/5737), [#5845](https://github.com/NVIDIA/Megatron-LM/pull/5845), [#5895](https://github.com/NVIDIA/Megatron-LM/pull/5895), [#5898](https://github.com/NVIDIA/Megatron-LM/pull/5898), [#6024](https://github.com/NVIDIA/Megatron-LM/pull/6024), [#6107](https://github.com/NVIDIA/Megatron-LM/pull/6107), [#6150](https://github.com/NVIDIA/Megatron-LM/pull/6150), [#6151](https://github.com/NVIDIA/Megatron-LM/pull/6151)

### DeepEP v2, Qwen3 Dynamic Decode, and Packed-THD MoE Correctness

MoE work is extending inference dispatch and decode performance while addressing heterogeneous expert layouts, packed-sequence auxiliary losses, expert-gradient topology, padding-aware expert-bias counts, fused activation mappings, interleaved layer validation, and NVFP4 execution under overlapped 1F1B.

- DeepEP v2 uses CUDA-graph-capturable expanded layouts and separate preallocated buffers for decode and chunked prefill.
- Sixteen opt-in Qwen3 MoE decode optimizations raised reported steady-state throughput from about 22.4k to 30,482 tokens per second on four GB200 GPUs.
- Heterogeneous expert counts, clamped SwiGLU, and corrected FlashInfer gated-weight mappings expand supported expert implementations.
- Packed THD auxiliary losses and expert parameter norms are being corrected across SP, CP, TP, EP, and differing expert-tensor-parallel topologies.

Related PRs: [#5601](https://github.com/NVIDIA/Megatron-LM/pull/5601), [#5745](https://github.com/NVIDIA/Megatron-LM/pull/5745), [#5900](https://github.com/NVIDIA/Megatron-LM/pull/5900), [#5940](https://github.com/NVIDIA/Megatron-LM/pull/5940), [#5996](https://github.com/NVIDIA/Megatron-LM/pull/5996), [#6045](https://github.com/NVIDIA/Megatron-LM/pull/6045), [#6064](https://github.com/NVIDIA/Megatron-LM/pull/6064), [#6079](https://github.com/NVIDIA/Megatron-LM/pull/6079), [#6080](https://github.com/NVIDIA/Megatron-LM/pull/6080), [#6114](https://github.com/NVIDIA/Megatron-LM/pull/6114), [#6115](https://github.com/NVIDIA/Megatron-LM/pull/6115), [#6135](https://github.com/NVIDIA/Megatron-LM/pull/6135), [#6165](https://github.com/NVIDIA/Megatron-LM/pull/6165)

### Optimizer Recipes, Selective FP32, Muon, and GTP Resharding

Optimizer and parameter-lifecycle work is adding declarative multi-optimizer recipes, selective FP32 execution, learned quantization scales, Muon support across layer-wise, tensor-parallel, and MFSDP v2 paths, plus GTP weight reuse, refit, resharding, and symmetric-memory collectives.

- YAML recipes define named optimizers, constructor arguments, parameter groups, glob matchers, defaults, and checkpoint-stable identities.
- Selective FP32 parameters retain their precision through execution and sharded optimizer checkpoints.
- Muon work addresses MXFP8 update propagation, high-padding layouts, small optimizer shards, per-head QKV partitioning, and adaptive-constructor reuse.
- GTP retains supported output-layer gathers for backward, adds native refit and resharding, and can register dense and routed-expert buffers for symmetric-memory collectives.

Related PRs: [#5707](https://github.com/NVIDIA/Megatron-LM/pull/5707), [#5738](https://github.com/NVIDIA/Megatron-LM/pull/5738), [#5818](https://github.com/NVIDIA/Megatron-LM/pull/5818), [#5925](https://github.com/NVIDIA/Megatron-LM/pull/5925), [#5929](https://github.com/NVIDIA/Megatron-LM/pull/5929), [#5986](https://github.com/NVIDIA/Megatron-LM/pull/5986), [#6015](https://github.com/NVIDIA/Megatron-LM/pull/6015), [#6068](https://github.com/NVIDIA/Megatron-LM/pull/6068), [#6101](https://github.com/NVIDIA/Megatron-LM/pull/6101), [#6133](https://github.com/NVIDIA/Megatron-LM/pull/6133), [#6147](https://github.com/NVIDIA/Megatron-LM/pull/6147), [#6155](https://github.com/NVIDIA/Megatron-LM/pull/6155)

### MIMO Placement, GPU-Resident RL Weight Sync, and Streaming Inference

Distributed post-training work is adding colocated and context-parallel non-colocated MIMO layouts, GPU-resident actor-to-vLLM weight synchronization, token-delta response streaming, prefix-cache lifecycle events, RL E-granularity, CP-aware sequence packing, cache-aware resume admission, and adaptive multi-environment concurrency.

- Colocated MIMO shares ranks while retaining separate parallel and RNG domains; non-colocated MIMO broadcasts embeddings and reduces gradients across destination TP×CP boundaries.
- GPU-resident, layout-aware synchronization bounds gathers and avoids CPU staging; the source implementation reported reducing update time from 84.88 seconds to 11.84–12.05 seconds on eight H100 GPUs.
- Dynamic inference streams generated token deltas and exposes cache lifecycle events for KV-aware routing.
- RL execution gains E-granularity, context-parallel packed sequences, and optional EMA-based redistribution of a fixed inference-concurrency budget.

Related PRs: [#5720](https://github.com/NVIDIA/Megatron-LM/pull/5720), [#5721](https://github.com/NVIDIA/Megatron-LM/pull/5721), [#5727](https://github.com/NVIDIA/Megatron-LM/pull/5727), [#5839](https://github.com/NVIDIA/Megatron-LM/pull/5839), [#5860](https://github.com/NVIDIA/Megatron-LM/pull/5860), [#5871](https://github.com/NVIDIA/Megatron-LM/pull/5871), [#5882](https://github.com/NVIDIA/Megatron-LM/pull/5882), [#5995](https://github.com/NVIDIA/Megatron-LM/pull/5995), [#6028](https://github.com/NVIDIA/Megatron-LM/pull/6028)

## Scope and Method

The narrative is grouped by technical theme rather than PR chronology. Related `dev` and `main` PRs are described once. The rebuildable raw ledger, exclusion evidence, commit activity, and state events are retained in the workflow database artifact. PRs closed without merge are counted but not narrated.