Megatron-LM Monthly Activity Report — September 2026 (through September 13, 2026)

# Megatron-LM Monthly Activity Report — September 2026 (through September 13, 2026)

> Chinese version: [中文版本](https://github.com/lhb8125/megatron-lm-activity-report/blob/main/reports/zh-CN/2026/09.md)

> Reporting window: 2026-09-01 through 2026-09-13 (Asia/Shanghai); source: [`NVIDIA/Megatron-LM`](https://github.com/NVIDIA/Megatron-LM).

## Overview

Delivered work expanded DeepSeek-V4 and hybrid-model capabilities, strengthened MoE and MFSDP execution, improved inference and context-parallel reliability, and added kernel determinism coverage. Ongoing work advances hybrid and multimodal architectures, distributed parallelism, sparse attention, checkpointing, CUDA Graph execution, inference, and training reliability.

- Substantive PR activity after filtering: opened 237; merged 115; closed without merge 59; active at cutoff 268.
- Excluded from the report and statistics: test-only 19, CI-only 6, test/CI-only 3, format-only 0.

## Delivered

### DeepSeek-V4 compressed sparse attention

DeepSeek-V4 received native compressed sparse attention orchestration, compact indexer execution, packed-THD fusion, and execution-aligned FLOP accounting.

- Native DeepSeek-V4 attention now constructs and schedules SBHD compressed sparse attention with module specifications, grouped outputs, MTP identity propagation, and indexer-loss tracking; compact cuDNN indexer and Top-K execution extend the path across BF16, MXFP8, CUDA graphs, and THD context parallelism. ([dev #5992](https://github.com/NVIDIA/Megatron-LM/pull/5992), [main #6402](https://github.com/NVIDIA/Megatron-LM/pull/6402))
- Packed-THD CSA execution fuses layout lowering, position mapping, indexer and compressor preparation, sparse-loss postprocessing, and related context-parallel kernels, while FLOP accounting covers sparse core attention, indexer work, absorbed-MLA costs, and hybrid DSA layers. ([dev #6753](https://github.com/NVIDIA/Megatron-LM/pull/6753), [main #7043](https://github.com/NVIDIA/Megatron-LM/pull/7043), [dev #7064](https://github.com/NVIDIA/Megatron-LM/pull/7064), [dev #7157](https://github.com/NVIDIA/Megatron-LM/pull/7157))

### Qwen3.5-VL HybridModel architecture

Qwen3.5-VL moved to an explicit hybrid architecture representation.

- The Qwen3.5-VL decoder now uses an explicit HybridModel stack of GatedDeltaNet, attention, dense, and MoE layers while preserving its layer cadence, MTP behavior, and mRoPE execution. ([dev #6315](https://github.com/NVIDIA/Megatron-LM/pull/6315))

### Inference coordinator routing

Inference coordination was updated for larger payloads, prefix-cache correctness, and load-aware routing.

- Inference coordinator wire framing, frontend prompt hashing, prefix-cache versioning and TTL handling, and load-aware routing were updated to move large payloads out of metadata, avoid stale cache matches, bound coordinator hash state, and balance cache affinity against fleet load. ([main #6497](https://github.com/NVIDIA/Megatron-LM/pull/6497))

### MoE fused execution, sharding, and precision

MoE execution gained alternate fused backends, independent dense and expert sharding, precise expert balancing, and safer precision overrides.

- MoE execution now exposes a pluggable megakernel boundary with a Mixture-of-Kittens backend, while MFSDP v2 allows dense and expert parameters to use independent optimizer-instance counts, meshes, and outer data-parallel sharding strategies. ([dev #6572](https://github.com/NVIDIA/Megatron-LM/pull/6572), [main #7012](https://github.com/NVIDIA/Megatron-LM/pull/7012))
- MoE expert token counters use int64 arithmetic for bias updates, and grouped-expert fusion is disabled when either linear is configured for non-quantized execution so precision overrides select the unfused path. ([main #7081](https://github.com/NVIDIA/Megatron-LM/pull/7081), [main #7212](https://github.com/NVIDIA/Megatron-LM/pull/7212))

### Runtime and packed-THD context parallelism

Context-parallel attention and packed-batch handling now use runtime state and preserve per-sequence layouts.

- Runtime context-parallel attention lazily creates the Transformer Engine CP stream and passes the runtime packed-sequence CP group to RoPE, ensuring attention uses the active stream and sequence-position partition. ([main #6821](https://github.com/NVIDIA/Megatron-LM/pull/6821))
- Packed THD batches now use multiple CP views and partition each packed sequence independently while reusing redistribution plans through MTP forward execution, keeping token shards and PackedSeqParams consistent. ([main #6912](https://github.com/NVIDIA/Megatron-LM/pull/6912))

### Distributed state and checkpoint reliability

Distributed weight copies, RNG restoration, and repeated MTP execution now preserve correct state across asynchronous and resumed workflows.

- Local refit-copy streams wait for the caller stream across NCCL, Gloo, NIXL, and NVSHMEM services before reading source weights, preventing same-rank copies from racing queued producer updates. ([main #6823](https://github.com/NVIDIA/Megatron-LM/pull/6823))
- Checkpoint RNG shard keys now include data-parallel and context-parallel rank, restoring independent expert-parallel RNG streams; repeated MTP layers also avoid passing stale first-microbatch state to Transformer Engine during checkpoint resume and backward execution. ([main #6858](https://github.com/NVIDIA/Megatron-LM/pull/6858), [main #6965](https://github.com/NVIDIA/Megatron-LM/pull/6965))

### MFSDP v2 CUDA graphs and gradient accumulation

MFSDP v2 gained full-iteration CUDA-graph support and correct no-sync gradient accumulation.

- MFSDP v2 now supports full-iteration CUDA graphs while preserving optimizer gradient bindings across replay; Hybrid mHC graph capture preserves fine-grained activation-offload events, callable backward-completion ordering, and rotary-position arguments so Transformer Engine replay retains offload dependencies and required RoPE inputs. ([dev #7014](https://github.com/NVIDIA/Megatron-LM/pull/7014), [main #7075](https://github.com/NVIDIA/Megatron-LM/pull/7075))
- MFSDP v2 no-sync handling marks non-final microbatches and installs the reduction context so accumulated gradients are finalized only on the last microbatch. ([main #7186](https://github.com/NVIDIA/Megatron-LM/pull/7186))

### Kernel determinism coverage

Kernel replay determinism is now enforced through byte-exact tests and CI coverage requirements.

- Byte-exact kernel replay tests, a kernel-source manifest, and a CI gate now require changed kernel files to carry determinism coverage or an approved exemption. ([main #7148](https://github.com/NVIDIA/Megatron-LM/pull/7148))

## In Progress

### Hybrid, DeepSeek, and multimodal model architectures

New model capabilities and execution paths extend HybridModel across DeepSeek, GLM, Engram, attention residual, audio, and multimodal workloads.

- HybridStack gains grouped scheduling with grouped layer allocation, expert-parallel overlap, recomputation, inference-cache indexing, and compatible checkpoint naming, while attention residuals add depth-wise softmax aggregation with pipeline-parallel, interleaved, hybrid-model, and MTP support. ([main #4942](https://github.com/NVIDIA/Megatron-LM/pull/4942), [dev #6840](https://github.com/NVIDIA/Megatron-LM/pull/6840))
- DeepSeek-V4 and V4.1 add sparse-attention construction and fused execution, NVFP4 training support, CSA2, cross-layer sharing, native mHC, THD, pipeline parallelism, recomputation, and CUDA-graph integration. ([main #6404](https://github.com/NVIDIA/Megatron-LM/pull/6404), [main #6841](https://github.com/NVIDIA/Megatron-LM/pull/6841), [dev #7224](https://github.com/NVIDIA/Megatron-LM/pull/7224), [dev #7285](https://github.com/NVIDIA/Megatron-LM/pull/7285))
- HybridModel adds ShortcutMoE, GLM-5.3 Flash, DeepSeek V4 Flash-Vision, and trainable Engram memory capabilities. ([main #6959](https://github.com/NVIDIA/Megatron-LM/pull/6959), [dev #7022](https://github.com/NVIDIA/Megatron-LM/pull/7022), [main #7054](https://github.com/NVIDIA/Megatron-LM/pull/7054), [main #7178](https://github.com/NVIDIA/Megatron-LM/pull/7178), [dev #7231](https://github.com/NVIDIA/Megatron-LM/pull/7231))
- Multimodal training expands with a dual-mode RoPE audio encoder, Energon conversation and media processing, and integrated packed multimodal SFT training. ([main #7066](https://github.com/NVIDIA/Megatron-LM/pull/7066), [main #7254](https://github.com/NVIDIA/Megatron-LM/pull/7254), [main #7255](https://github.com/NVIDIA/Megatron-LM/pull/7255))

### Checkpoint merging, interoperability, and trainable-parameter state

Checkpointing work improves distributed merging, optimizer resharding, GPT-to-Hybrid compatibility, logical GTP shard handling, and adapter-only saves.

- A CPU-only metadata-driven tool merges compatible torch_dist checkpoints with weighted and WSM policies, external common state, dry runs, and explicit state filtering. ([main #5114](https://github.com/NVIDIA/Megatron-LM/pull/5114))
- MFSDP v2 adds optimizer checkpoint save/load in fsdp_dtensor DCP format, including rank-consistent placeholders for empty local shards. ([main #6215](https://github.com/NVIDIA/Megatron-LM/pull/6215))
- Bridge-compatible train_state sidecars and GPT-to-Hybrid conversion improve restoration and interoperability of model, optimizer, and training state, while trainable-parameter filtering supports adapter-only PEFT/LoRA checkpoints that restore trainable parameters, optimizer state, and iteration state without frozen base weights. ([main #6807](https://github.com/NVIDIA/Megatron-LM/pull/6807), [pull-request/4942 #7192](https://github.com/NVIDIA/Megatron-LM/pull/7192), [main #7280](https://github.com/NVIDIA/Megatron-LM/pull/7280))
- GTP checkpointing now handles fused projection reconstruction, reshardable optimizer state, logical padding removal, optimizer identity, and grouped gated projections. ([main #7161](https://github.com/NVIDIA/Megatron-LM/pull/7161), [main #7162](https://github.com/NVIDIA/Megatron-LM/pull/7162), [main #7168](https://github.com/NVIDIA/Megatron-LM/pull/7168))

### THD, DSA, and sparse context-parallel execution

Packed-THD and DSA work adds balanced indexing, explicit planning, min-memory kernels, training integration, and graph-safe sparse attention.

- DSA indexing and scoring add balanced context-parallel routing, explicit THD plans, host-prebuilt layout tables, static-shape backward handling, and stable metric reduction. ([dev #6058](https://github.com/NVIDIA/Megatron-LM/pull/6058), [dev #6206](https://github.com/NVIDIA/Megatron-LM/pull/6206), [pull-request/6206 #6910](https://github.com/NVIDIA/Megatron-LM/pull/6910), [dev #6980](https://github.com/NVIDIA/Megatron-LM/pull/6980))
- MTP and hybrid context-parallel paths add absolute-offset sequence rolls, contiguous-layout conversion through GDN, and packed-THD batching with per-sequence zigzag padding. ([pull-request/6473 #6741](https://github.com/NVIDIA/Megatron-LM/pull/6741), [main #7128](https://github.com/NVIDIA/Megatron-LM/pull/7128), [main #7238](https://github.com/NVIDIA/Megatron-LM/pull/7238))
- Simplified DSA over GQA gains min-memory Triton and autograd kernels, an integrated sparse-attention layer, indexer training controls, and DSA-over-GQA inference. ([main #6869](https://github.com/NVIDIA/Megatron-LM/pull/6869), [main #7152](https://github.com/NVIDIA/Megatron-LM/pull/7152), [pull-request/7152 #7156](https://github.com/NVIDIA/Megatron-LM/pull/7156), [pull-request/7158 #7181](https://github.com/NVIDIA/Megatron-LM/pull/7181), [main #7211](https://github.com/NVIDIA/Megatron-LM/pull/7211))
- Fused CSA RoPE bounds and top-k selection are hardened for safe and repeatable sparse-attention execution. ([main #7104](https://github.com/NVIDIA/Megatron-LM/pull/7104), [dev #7169](https://github.com/NVIDIA/Megatron-LM/pull/7169))

### MFSDP v2, MXFP8, and fused distributed gradients

MFSDP v2 work combines quantized weights, pipeline and 1F1B scheduling, fused gradient paths, and distributed communication overlap.

- An MFSDP v2 prototype composes MXFP8 primary weights, virtual pipeline parallelism, combined 1F1B scheduling, and distributed Muon on a hybrid MoE workload. ([main #6197](https://github.com/NVIDIA/Megatron-LM/pull/6197))
- MFSDP v2 adds MXFP8 parameter groups with persistent FP32 masters and row- and column-wise payload buffers. ([main #6485](https://github.com/NVIDIA/Megatron-LM/pull/6485))
- Fused weight-gradient accumulation writes directly into MFSDP v2 reduce-scatter buffers, while fine-grained 1F1B integrates parameter lifecycle hooks and no-sync gradient accumulation. ([main #7101](https://github.com/NVIDIA/Megatron-LM/pull/7101), [main #7112](https://github.com/NVIDIA/Megatron-LM/pull/7112))

### Dynamic context parallelism and explicit process groups

Parallel-state modernization and dynamic CP provide runtime topology selection across model, schedule, MoE, and hybrid consumers.

- Core model and attention paths increasingly consume explicit ProcessGroupCollection instances instead of global parallel-state accessors. ([main #6293](https://github.com/NVIDIA/Megatron-LM/pull/6293))
- Dynamic CP adds runtime group resolution, singleton CP1 handling, arbitrary CP sizes, exact validation, and exception-safe Transformer Engine binding. ([main #7045](https://github.com/NVIDIA/Megatron-LM/pull/7045), [dev #7097](https://github.com/NVIDIA/Megatron-LM/pull/7097))
- Dynamic CP integrates with training and evaluation schedules, packed-microbatch MoE losses, and hybrid layers including Mamba, GDN, MLA, and DSA. ([main #7046](https://github.com/NVIDIA/Megatron-LM/pull/7046), [main #7047](https://github.com/NVIDIA/Megatron-LM/pull/7047), [main #7048](https://github.com/NVIDIA/Megatron-LM/pull/7048))

### CUDA Graph capture for dynamic and hybrid training

CUDA Graph work expands capture coverage while supporting dynamic packed inputs, hybrid execution, offload, and pipeline schedules.

- Dynamic CP graph execution shares slot and liveness arenas across graph banks, reducing duplicated storage while retaining per-size executables and process groups. ([dev #6479](https://github.com/NVIDIA/Megatron-LM/pull/6479))
- Full-iteration and partial HybridStack capture support grouped layers, paged MoE stashing, graphable Mamba and attention spans, and eager dynamic expert execution. ([main #6958](https://github.com/NVIDIA/Megatron-LM/pull/6958), [main #6994](https://github.com/NVIDIA/Megatron-LM/pull/6994))
- Capture-safe offload and dynamic packed inputs separate local and external events and stage changing THD and balanced-indexer metadata into fixed-address replay buffers. ([dev #7020](https://github.com/NVIDIA/Megatron-LM/pull/7020), [dev #7102](https://github.com/NVIDIA/Megatron-LM/pull/7102))
- Balanced DSA graphs extend to EP 1F1B overlap, pipeline and virtual-pipeline schedules, and dynamic packed microbatches. ([dev #7206](https://github.com/NVIDIA/Megatron-LM/pull/7206))

### Disaggregated and dynamic inference

Inference work improves prefill/decode disaggregation, MTP caching, prefix-score reuse, request lifecycle handling, and generation compatibility.

- Prefill/decode disaggregation transfers live KV and SSM state through coordinator-managed handoff using NCCL or NIXL. ([main #6675](https://github.com/NVIDIA/Megatron-LM/pull/6675))
- Inference correctness improvements cover NVLS row binding, resume rebinding, inherited KV blocks, deterministic greedy sampling, and complete EOS and sampling-default handling. ([main #7061](https://github.com/NVIDIA/Megatron-LM/pull/7061), [main #7191](https://github.com/NVIDIA/Megatron-LM/pull/7191))
- MTP inference adds dynamic-context KV caching, while prefix caching stores and reconstructs prompt log probabilities from canonical KV blocks. ([main #7033](https://github.com/NVIDIA/Megatron-LM/pull/7033), [main #7063](https://github.com/NVIDIA/Megatron-LM/pull/7063))
- Dynamic inference hardens admission, pause/resume, recomputation, completion, reset, failure ownership, prompt-memory release, and token-score handling; gathered MoE weights are refreshed in place for serving and resume. ([main #6931](https://github.com/NVIDIA/Megatron-LM/pull/6931), [main #7256](https://github.com/NVIDIA/Megatron-LM/pull/7256))

### Muon, GTP, GDN, and distributed training reliability

Optimizer, fused-kernel, communication, and runtime-safety changes improve distributed execution and reduce failure modes.

- Muon adds compact LayerWise layouts, FP8 gathering, per-head QKV orthogonalization, layer-sharded updates, generalized tensor-parallel integration, and multimodal Qwen3.5-VL support. ([main #5479](https://github.com/NVIDIA/Megatron-LM/pull/5479), [main #6326](https://github.com/NVIDIA/Megatron-LM/pull/6326), [dev #6608](https://github.com/NVIDIA/Megatron-LM/pull/6608), [main #6683](https://github.com/NVIDIA/Megatron-LM/pull/6683), [dev #6882](https://github.com/NVIDIA/Megatron-LM/pull/6882), [dev #7085](https://github.com/NVIDIA/Megatron-LM/pull/7085))
- GatedDeltaNet gains an internal CuTe backend with FLA fallback, while EP-overlap scheduling supports segmented recomputation with RNG and MTP state snapshots. ([main #6429](https://github.com/NVIDIA/Megatron-LM/pull/6429), [dev #6511](https://github.com/NVIDIA/Megatron-LM/pull/6511))
- Virtual-expert MoE routing adds replica exchange and gradient-reduction paths, and MFSDP/FSDP runtime changes improve NCCL pool ownership and model-parallel teardown. ([main #6892](https://github.com/NVIDIA/Megatron-LM/pull/6892), [main #6923](https://github.com/NVIDIA/Megatron-LM/pull/6923), [main #7029](https://github.com/NVIDIA/Megatron-LM/pull/7029))
- Training diagnostics and optimizer control improve through reduced host synchronization, per-group optimizer clocks, custom quantization recipes, mixed-dtype gradient clipping, and distributed edge-case handling. ([main #7076](https://github.com/NVIDIA/Megatron-LM/pull/7076), [pull-request/7156 #7158](https://github.com/NVIDIA/Megatron-LM/pull/7158), [main #7176](https://github.com/NVIDIA/Megatron-LM/pull/7176), [main #7177](https://github.com/NVIDIA/Megatron-LM/pull/7177), [main #7279](https://github.com/NVIDIA/Megatron-LM/pull/7279))

### NCCL resharding and large-scale transfer safety

Communication reliability work constrains large transfers to avoid dependency cycles and deadlocks during refit and resharding.

- NCCL resharding P2P submissions are bounded with parameter batches and deterministic task-id windows for oversized transfer groups. ([main #7189](https://github.com/NVIDIA/Megatron-LM/pull/7189))

## Scope and Method

The narrative is grouped by technical theme rather than PR chronology. Related `dev` and `main` PRs are described once. The rebuildable raw ledger, exclusion evidence, commit activity, and state events are retained in the workflow database artifact. PRs closed without merge are counted but not narrated. Each PR citation identifies its target branch. Open PRs do not carry into a new month without code or state activity in that month.

The issue contains a generated September 2026 activity report and names no source file, test, or implementation entry point. Start by reviewing the report body and its source marker; there is no requested code or documentation change, so completion cannot be determined from this issue alone.

Written by the indexing model from the issue text.