GLM-5.2 training support tracking

# GLM-5.2 training support tracking

> End-to-end GLM-5.2 training support, covering model fidelity, DSA/IndexShare, repeated MTP, Muon, BF16/MXFP8, long-context THD training, memory safety, and production performance on Hopper and Blackwell.

Last updated: 2026-09-07

Status: ✅ merged/integrated · ↧ merged to `main` and awaiting synchronization · 🚧 open/in review · 🧪 prototype validated but not upstream · 📋 planned · ❓ behavior not yet established

Priority: **P0** release or full-model blocker · **P1** performance or important non-blocking functionality · **P2** on-demand capability or structural cleanup

Ordering within each theme: priority first (`P0` → `P2`), then progress (completed/synchronized → active → prototype → planned → unknown).

## Scope and completion criteria

This roadmap uses the latest Megatron-LM `dev` as its development, integration, and validation baseline. Contributions targeting `main` remain valid dependencies and are labeled `main`; they enter the working baseline through the regular upstream synchronization flow. Experimental integration branches may combine pending PRs for end-to-end testing, but are not delivery artifacts.

The roadmap is complete when:

- MCore reproduces every behavior unambiguously specified by the public GLM-5.2 config and checkpoint.
- BF16 works on H100, GB200, and GB300; MXFP8 works on GB200 and GB300.
- THD, dynamic/static CP, PP, EP, DSA, IndexShare, repeated MTP, Muon, recompute, and CUDA Graph compose correctly.
- GB200 and GB300 have reproducible proxy and full-model performance, memory, and Nsight Systems evidence.
- Checkpoint resume and short convergence pass with the intended optimizer and production data path.

References:

- [GLM-5.2 model and public config](https://huggingface.co/zai-org/GLM-5.2)
- [GLM-5.2 architecture and RL release notes](https://huggingface.co/blog/zai-org/glm-52-blog)
- [GLM-5 technical report](https://arxiv.org/html/2602.15763)
- [Bebop end-to-end TV objective](https://arxiv.org/html/2606.12370)
- [IndexCache](https://arxiv.org/html/2603.12201)
- [Megatron Core Q3 roadmap](https://github.com/NVIDIA/Megatron-LM/issues/5676)

## Overall status

Ordinary DSA is integrated on `dev`, including the production cuDNN path, THD, CP, layer-wise IndexShare, latent/indexer precision controls, standard MLA/DSA RoPE fusion, Whole-MoE paged-stash CUDA Graph support, and the base DSA FLOP-accounting model. GLM-5.2 is **integration-capable but not yet production-qualified**. P0 gaps remain in exact optimizer fidelity, CP1 fused indexer execution, and long-context host overhead; sparse indexer-loss backward still needs execution-aware FLOP accounting. Stage-aware training and RL-safe DSA execution remain P2 lifecycle capabilities. Generic repeated-MTP execution is available, while cross-iteration IndexShare/KVShare, its activation-recompute composition, and end-to-end TV have open implementations. The optional MOK MoE megakernel backend has landed on `dev`, but GLM-5.2 qualification remains P2 work. The aggregate integration stack has produced formal fixed-shape 256×GB300 MXFP8 performance and memory evidence for repeated MTP7 and TV, but that evidence does not replace source-PR review or establish GB200, BF16, checkpoint, convergence, or iteration-varying dynamic-CP qualification. The exact TV coefficient, training budget, and phase placement remain undisclosed.

## Landed baseline

Capability | Status | Evidence
--- | --- | ---
Ordinary DSA with cuDNN, THD, CP, and layer-wise IndexShare | ✅ | Main implementation from [#5099](https://github.com/NVIDIA/Megatron-LM/pull/5099) and [#5243](https://github.com/NVIDIA/Megatron-LM/pull/5243)–[#5246](https://github.com/NVIDIA/Megatron-LM/pull/5246), integrated by [#6020](https://github.com/NVIDIA/Megatron-LM/pull/6020).
Attention latent-norm and indexer projection-precision controls | ✅ | Integrated on `dev` through [#6204](https://github.com/NVIDIA/Megatron-LM/pull/6204) and [#6279](https://github.com/NVIDIA/Megatron-LM/pull/6279).
Fused standard MLA/DSA RoPE packing | ✅ | Integrated on `dev` through [#6343](https://github.com/NVIDIA/Megatron-LM/pull/6343).
Whole-MoE paged-stash CUDA Graph support | ✅ | Integrated on `dev` through [#6022](https://github.com/NVIDIA/Megatron-LM/pull/6022).
Base DSA FLOP accounting | ✅ | Integrated on `dev` through [#6753](https://github.com/NVIDIA/Megatron-LM/pull/6753).
Per-CP-lane DP all-gather for balanced THD rerouting | ✅ | Integrated on `dev` through [#6378](https://github.com/NVIDIA/Megatron-LM/pull/6378); production full-model validation remains open.
Optional MOK MoE megakernel backend | ✅ | Integrated on `dev` through [#6572](https://github.com/NVIDIA/Megatron-LM/pull/6572); GLM-5.2-specific qualification is tracked below.

## Functionality and model parity

> **Theme status:** The ordinary DSA architecture and latent/indexer precision controls are available, but end-to-end GLM-5.2 training equivalence is not yet established. The immediate model-fidelity blocker is per-head Muon treatment for the publicly specified `W^UQ`, `W^UK`, and `W^UV` matrices without inventing semantics for `W^QR` or packed down projections; stage-aware training remains P2 lifecycle work. Generic repeated-MTP execution exists, while cross-iteration sharing and the GLM-5.2 end-to-end TV objective have open PRs plus aggregate full-model GB300 execution evidence. The remaining unknowns include the exact TV coefficient and adaptation budget, not whether GLM-5.2 uses TV.

### Numerical fidelity and precision

Priority | Status | Work item | Current finding | Next gate | Tracking
--- | --- | --- | --- | --- | ---
P0 | ✅ | Attention latent-norm epsilon | [#6204](https://github.com/NVIDIA/Megatron-LM/pull/6204) landed the independent attention-latent epsilon needed to combine transformer-block RMSNorm at `1e-5` with Q/KV latent norms at `1e-6`; indexer-K remains controlled separately at `1e-6`. | Retain fused/unfused, training/inference, RMSNorm/LayerNorm orthogonality, and DSA/DSv4 parity while qualifying the complete GLM configuration. | [#6204](https://github.com/NVIDIA/Megatron-LM/pull/6204).
P0 | ✅ | Indexer projection precision control | [#6279](https://github.com/NVIDIA/Megatron-LM/pull/6279) landed an explicit output-precision policy, allowing the released FP8 checkpoint's BF16 `indexer.weights_proj` behavior without relying on module-name accidents. Whether BF16 output is a training requirement or an HF inference-stability choice remains unconfirmed. | Validate BF16/MXFP8 value, top-k, loss, gradient, and checkpoint behavior in the complete recipe; keep the unresolved training interpretation explicit. | [#6279](https://github.com/NVIDIA/Megatron-LM/pull/6279).

### Optimizer parity

Priority | Status | Work item | Current finding | Next gate | Tracking
--- | --- | --- | --- | --- | ---
P0 | 🚧 | Per-head Muon for GLM attention projections and packed MLA layout | The GLM-5 report requires independent orthogonalization and update scales for each attention head of `W^UQ`, `W^UK`, and `W^UV`. In MLA notation, `W^UQ` is the no-PE query up-projection; the report does not establish the optimizer treatment of `W^QR` or any down-projection split. #6326 adds module-owned packed layouts, per-head orthogonalization, and fragmented TP/GTP reconstruction for MLA, FusedMLA, and AbsorbedMLA, superseding the overlapping up-projection boundary tagging in #5015. Its current MLA Q layout also applies per-head orthogonalization to `W^QR`, which goes beyond the publicly established GLM behavior. The aggregate full-model matrix executed the per-head option successfully but does not establish official optimizer parity. | Refine the packed layout policy so every head of `W^UQ`, `W^UK`, and `W^UV` receives independent Newton–Schulz/scaling without assuming an unpublished `W^QR` policy. Validate against a batched per-head native reference, then close TP/GTP execution, optimizer-step performance, optimizer-state save/load, BF16/MXFP8, and fused/unfused numerical parity. | [#6326](https://github.com/NVIDIA/Megatron-LM/pull/6326).
P1 | ❓ | Unresolved optimizer treatment for `W^QR` and packed down-projections | Public GLM material specifies per-head Muon Split for `W^UQ`, `W^UK`, and `W^UV`, but does not establish the optimizer assignment for `W^QR` or whether physically packed `W^DQ`, `W^DKV`, and `W^KR` should be split into logical matrices. Physical fusion must not silently determine optimizer semantics, and neither #5015 nor #6326 establishes an authoritative policy for these matrices. | Establish the intended optimizer and grouping for each logical matrix, then encode it independently of packed storage. Validate fused/unfused one-step updates, optimizer-state checkpoint/resume, TP/distributed execution, and BF16/MXFP8 parity. | TBD.
P1 | ❓ | GLM Muon hyperparameters and remaining parameter grouping | The published GLM-5 lineage uses Newton–Schulz `N=5`, momentum `0.95`, update RMS `0.2`, weight decay `0.1`, and excludes word embeddings, biases, and RMSNorm weights from Muon. Exact GLM-5.2 inheritance is not established, and the public material does not disclose the exact Nesterov setting, Newton–Schulz coefficients, fallback optimizer and its betas/epsilon, gradient clipping, complete weight-decay exclusions, LM-head treatment, or fused-SwiGLU gate/up grouping. A runnable generic-Muon recipe therefore cannot yet claim optimizer parity. | Separate published values from explicit MCore assumptions; obtain or document the unresolved settings; audit every remaining parameter group and exclusion; then validate optimizer-state checkpoint/resume, BF16/MXFP8, and short convergence. | TBD.

### Model execution and pipeline scheduling

Priority | Status | Work item | Current finding | Next gate | Tracking
--- | --- | --- | --- | --- | ---
P1 | 📋 | 1F1B overlap support validation | The individual pipeline and communication-overlap mechanisms exist, but their production composition with GLM-5.2 DSA, repeated MTP, deferred wgrad, PP, and distributed optimizer has not been validated as a complete configuration. | Compare overlap on/off under the same 1F1B schedule; require loss and gradient parity, correct deferred-wgrad flush ordering, checkpoint-resume equivalence, and measured communication exposure. | TBD.
P2 | 📋 | MTP-aware pipeline-layout balance | The selected MTP-on PP4 layout `Et*18\|t*24\|t*24\|t*12m*7L` is runnable, but rank-sampled 128K NSys shows estimated logical work shares of 19.1%/25.6%/25.6%/29.7% across PP0–PP3. PP3 carries the repeated MTP7 vocabulary projection and TV objective in addition to transformer layers, while PP0 also exposes a larger pipeline send/receive share; the current layout is therefore validated for execution, not established as throughput-optimal. | Build a stage cost and memory model that includes DSA cadence, MoE layers, output/loss work, and repeated-MTP depth. Evaluate candidate MTP-on and MTP-off layouts under identical topology, 32-microbatch schedule, and model semantics; require correctness first, then no-profiler ABBA plus rank-sampled NSys to minimize rank-max step time without violating HBM margin. | Aggregate evidence [#6944](https://github.com/NVIDIA/Megatron-LM/pull/6944).
P2 | 📋 | GLM-5.2 HybridModel execution path | HybridModel can assemble AbsorbedMLA and ordinary DSA, but the current GLM-5.2 recipes still build GPTModel. HybridStack numbers split attention and MLP symbols independently, while GLM-5.2 IndexShare, PP boundaries, and repeated-MTP sharing require logical transformer-layer numbering. | Define the 3-dense/75-MoE HybridModel pattern and a stable logical-layer contract; then prove GPTModel/HybridModel parameter, checkpoint, forward/backward, loss, IndexShare, PP, and repeated-MTP parity. | TBD.

### Training metrics

Priority | Status | Work item | Current finding | Next gate | Tracking
--- | --- | --- | --- | --- | ---
P0 | ✅ | Base DSA FLOPs and TFLOPS | [#6753](https://github.com/NVIDIA/Megatron-LM/pull/6753) replaces dense-MLA core-attention cost with the executed top-k-scaled absorbed-MLA form, adds indexer projection and dense scoring cost, respects IndexShare compute-layer frequency and offset, and distinguishes the loss-off and loss-on forward/backward expansions. This is a precision-independent logical-work convention: every multiply-add is two FLOPs whether executed in FP32, BF16, or MXFP8. It makes the pinned Megatron source convention usable, but is neither precision-weighted hardware throughput nor a hardware-instruction counter. | Retain exact-shape golden coverage and update the convention whenever DSA execution semantics change. Treat execution-specific replay, recompute, and sparse scoring-backward refinements separately. | [#6753](https://github.com/NVIDIA/Megatron-LM/pull/6753).
P1 | 🚧 | Indexer-loss metric aggregation under dynamic CP and CUDA Graph | Detached DSA/CSA indexer loss can be produced by only a subset of layers and ranks. [#6980](https://github.com/NVIDIA/Megatron-LM/pull/6980) makes its logging ownership explicit across dynamic CP, PP, Hybrid/repeated-MTP execution, and CUDA Graph: it reduces over the stable parent DP×CP group, identifies actual loss-owner layers and owning PP/DP groups, preallocates tracker storage, and removes capture-only, stale, or double-counted contributions. This changes metric aggregation, not attention numerics, backward execution, or FLOP accounting. | Complete review and merge; validate exact per-layer and global metrics against an independent reduction oracle across CP1/CP>1, changing effective CP, PP, Hybrid/repeated MTP, and capture/replay. | [#6980](https://github.com/NVIDIA/Megatron-LM/pull/6980).
P1 | 🧪 | Sparse indexer-loss scoring-backward FLOP accounting | When `dsa_indexer_use_sparse_loss` is enabled, the upstream source convention still charges the indexer scoring GEMM as dense causal forward plus dense dQ/dK (`3×` the dense scoring term). The executed cuDNN split-DSA path keeps scoring forward dense but restricts its backward scoring GEMMs to selected causal top-k pairs. The runner-only full-model audit uses exact per-sequence causal-pair counts, charges dense forward plus sparse-pair backward-path GEMMs, and includes the split-path replay GEMMs required by the resolved execution contract. That audited logical metric is not upstream Megatron accounting and is not a hardware-instruction counter. | Add execution-aware upstream accounting that preserves dense scoring-forward cost while charging realized backward/recompute GEMMs from exact `sum_i min(i, topk)` causal pairs rather than dense `sum(L²)`. Cover sparse-loss on/off, ragged THD, IndexShare frequency/offset, CP1/CP>1 and dynamic CP, repeated MTP, and split/fused backends; validate with an independent golden calculator and operator-level traces. | Landed foundation [#6753](https://github.com/NVIDIA/Megatron-LM/pull/6753); upstream follow-up TBD.

### DSA and repeated MTP semantics

Priority | Status | Work item | Current finding | Next gate | Tracking
--- | --- | --- | --- | --- | ---
P0 | 🚧 | GLM-5.2 end-to-end TV objective | The GLM-5.2 release explicitly states that MTP is trained with end-to-end TV loss and uses rejection sampling for speculative decoding; this is not an optional parity item. #6473 implements Bebop-style online TV with detached target heads, TP-aware full-vocabulary distance, packed-THD masking, and analytical draft-only backward. Cross-entropy may remain the generic MCore default, but the GLM-5.2 recipe must select TV. The coefficient, data mixture, duration, and exact phase placement remain undisclosed. The aggregate stack has executed repeated MTP7 with E2E-TV in the formal 256×GB300 4K/128K MXFP8 matrix; this is integration evidence, not source-head merge readiness. | Complete source-PR review and CI; make TV explicit in delivery recipes; then close source-head checkpoint/resume, BF16 and GB200 qualification, short convergence, and acceptance quality without inventing an authoritative coefficient. | [#6473](https://github.com/NVIDIA/Megatron-LM/pull/6473).
P1 | 🚧 | MTP-iteration IndexShare and KVShare | [#6472](https://github.com/NVIDIA/Megatron-LM/pull/6472) reuses depth-0 latent KV and/or top-k at later repeated-MTP depths while preserving query-dependent sparse attention. Its refreshed source head covers independent and combined sharing, CP1/CP2 and dynamic CP, TP/SP layouts, PP/VPP placement, and forward/gradient parity. Activation recompute is intentionally out of scope and fails closed whenever `recompute_granularity` is enabled. The aggregate full-model results used an earlier recompute-inclusive integration snapshot and therefore do not establish the current #6472 source head. | Complete fresh GPU CI and source review, then close no-recompute source-head loss/output/gradient, checkpoint/resume, BF16/GB200, and production attention CUDA Graph composition. Preserve the explicit seven-step delivery recipe without attributing recompute support to #6472. | Open [#6472](https://github.com/NVIDIA/Megatron-LM/pull/6472).
P1 | 🚧 | Activation recompute with repeated-MTP sharing | Dependent draft [#7107](https://github.com/NVIDIA/Megatron-LM/pull/7107) owns full and selective recompute support. It carries depth-0 shared tensors through checkpoint replay, keeps shared latent KV outside selective MLA up-projection recompute, and snapshots ordinary IndexShare state. It still reuses the cross-depth sharing context as replay owner, lacks dedicated ordinary-IndexShare coverage with cross-depth sharing disabled, and awaits fresh GPU validation. | Separate replay ownership from cross-depth tensor transport; add ordinary IndexShare coverage; rebase onto `dev` after #6472 lands; then validate full/selective recompute output, loss, and gradient parity, stale-state isolation, checkpoint/resume, memory, and performance. | Base [#6472](https://github.com/NVIDIA/Megatron-LM/pull/6472); dependent Draft [#7107](https://github.com/NVIDIA/Megatron-LM/pull/7107).
P2 | ❓ | IndexShare multi-layer distillation | IndexCache proposes multi-layer guidance for a shared indexer. GLM-5.2 publicly confirms four-layer IndexShare, but no public evidence establishes that the released model used this additional distillation loss. It is a potential IndexShare quality/convergence improvement, not a training stage or established parity requirement. | First reproduce the reported benefit against the existing per-indexer loss with an independent reference; if retained, add it behind an explicit objective option and validate layer weighting, IndexShare boundaries, loss scaling, gradients, checkpoint resume, and short convergence. | TBD.

### Training stages and objectives

This section tracks only MCore training-control capabilities. MTP-TV is tracked above as a model objective; SFT/RL algorithms, rollout orchestration, OPD, and unpublished GLM-5.2 phase budgets are outside this table. In the published GLM-5 lineage, DSA is introduced after dense mid-training, so the concrete flow is dense pre-/mid-training → indexer-only warmup → joint sparse adaptation.

Priority | Status | Work item | Current finding | Next gate | Tracking
--- | --- | --- | --- | --- | ---
P2 | ❓ | Smooth dense-training → indexer-warmup → joint-DSA transition | MCore can run the component objectives separately, but the complete handoff is not a first-class, validated workflow. The transition must preserve model and optimizer state, LR schedule, consumed samples/tokens, RNG/data state, IndexShare configuration, and checkpoint compatibility. | Provide stage-aware recipes and transition/resume support; compare uninterrupted execution with save/exit/relaunch at both boundaries, including loss, gradients, parameter/optimizer state, and counters. | TBD.
P2 | ❓ | Indexer-only training mode | The published DSA warmup freezes all base-model weights and trains only the indexer for 1,000 steps before joint sparse adaptation. MCore lacks an explicit mode that defines and audits this trainable-parameter boundary. Whether GLM-5.2 repeated the warmup when adding IndexShare remains undisclosed. | Add an explicit mode that enables only indexer parameters, excludes frozen parameters from optimizer-state allocation, and validates zero backbone gradients, loss scaling, checkpoint round trip, and transition into joint training. | TBD.
P2 | ❓ | RL-safe DSA indexer execution | The GLM-5 report states that its RL training engine uses deterministic naive `torch.topk` and freezes indexer parameters by default. The same indexer still runs forward to select sparse-attention positions; replaying all `k=2048` indices from rollout is rejected as too expensive. Public material does not disclose the final rollout-side top-k implementation or whether auxiliary indexer loss remains enabled, and GLM-5.2 has not separately restated this policy. Performance-only force-balanced MoE routing is also invalid for RL correctness. | Add explicit, independent controls for indexer trainability, training-side top-k implementation, and auxiliary-loss policy. Reproduce the disclosed GLM-5 behavior by removing frozen indexer parameters from optimizer state and selecting deterministic training-side top-k; make the rollout backend and auxiliary-loss choice explicit rather than assuming unpublished defaults. Validate train/rollout selected-index and policy-log-prob mismatch, zero indexer updates, unchanged weights, no force balancing, and checkpoint resume. | TBD.

Published context retained outside the tracking table:

- **MTP objective placement:** GLM-5.2 directly specifies shared repeated-MTP parameters, first-step IndexShare/KVShare, rejection sampling, and end-to-end TV training. These model semantics are tracked in the preceding section. The TV coefficient, data budget, duration, and placement within base or post-training remain unpublished; there is no evidence for a standalone MTP-adaptation stage.
- **SFT and RL sequence:** The GLM-5 lineage proceeds through multi-task SFT, Reasoning RL, Agentic RL, General RL, and on-policy cross-stage distillation. This is useful checkpoint and handoff context, but the complete GLM-5.2 stage order, loss mixture, optimizer-reset policy, and budgets are not public.
- **Reasoning-RL lineage settings:** GLM-5 reports GRPO/IcePop-style fully on-policy training without a KL term, with mismatch bound `beta=2`, clipping `epsilon_low=0.2`/`epsilon_high=0.28`, group size 32, and batch size 32. These are GLM-5 lineage values, not confirmed GLM-5.2 configuration.
- **GLM-5.2 long-horizon post-training:** The GLM-5.2 release directly describes critic-based PPO over individual rollouts, token-level advantages, training on all compacted sub-traces with token-level loss, online rule-plus-LLM anti-hack handling, and parallel OPD over more than ten expert models. These belong to the external RL/rollout stack rather than the three MCore DSA training-control items above.

## Performance

> **Theme status:** cuDNN is the working production baseline, TileLang is available on `dev`, and the invalid-slot indexer-backward optimization has landed. A newer generic SM100 cuDNN DSA backward kernel is merged upstream but not yet present in the measured GLM image, and the landed MOK backend has not yet been qualified for GLM-5.2. CP1 can still miss the intended fused scorer, long-context attention retains substantial host-launch overhead, and cuDNN-versus-TileLang qualification remains open. The aggregate matrix establishes fixed-shape, fixed-microbatch full-iteration CUDA Graph gains on GB300; iteration-varying dynamic CP and microbatch counts still require the DSA graph-safety plus chunk-wise/dynamic-CP graph infrastructure.

### DSA/indexer kernels and backends

Priority | Status | Work item | Current finding | Next gate | Tracking
--- | --- | --- | --- | --- | ---
P0 | 🚧 | General packed-THD fused indexer scorer | #6206 has an open implementation that selects scoring from actual packed metadata rather than CP size, runs multi-sequence CP1 as one cuDNN THD call, and supports CP>1 packed scoring with query-position and gathered-key reorder metadata. It retains conservative unfused fallback when backend preconditions cannot be proven. | Complete review and merge of the current #6206 head; require value/top-k/gradient parity across CP1/CP>1, single/multiple/empty/padded segments and dynamic CP, then measure scorer dispatch, fallback rate, memory, and end-to-end performance on the integrated stack. | Open [#6206](https://github.com/NVIDIA/Megatron-LM/pull/6206).
P1 | ✅ | Indexer backward performance | Profiling exposed excessive sparse-indexer backward work on invalid and block-padding slots: they carried index `0` with zero score, which was numerically correct but still caused unnecessary K loads and zero-valued dK atomics. #6166 preserves the `-1` sentinel so the kernel's bounds guard skips that work. | Completed: retain invalid-slot parity coverage and benchmark the path when changing the sparse backward kernel. | [#6166](https://github.com/NVIDIA/Megatron-LM/pull/6166).
P1 | 🚧 | cuDNN versus TileLang | cuDNN is the production recipe backend. TileLang and the GLM indexer projection-precision controls are available on `dev` through #6279, but no controlled current-stack backend comparison has established its production value. | 16/64-GPU short/long-sequence A/B with identical configs, memory, and rank-sampled NSys. | `dev` merged [#6279](https://github.com/NVIDIA/Megatron-LM/pull/6279); original `main` implementation [#5049](https://github.com/NVIDIA/Megatron-LM/pull/5049).
P2 | 📋 | Updated cuDNN DSA sparse-attention backward qualification | Sparse-attention backward remains a material kernel cost. [cuDNN Frontend #684](https://github.com/NVIDIA/cudnn-frontend/pull/684) is merged and included in v1.28.0, but the measured GLM image predates it. For GLM-5.2's exact `H=64, Dqk=576, Dv=512, top-k=2048` shape, upstream B200 operator data reports `6.049→5.248 ms` (`1.153×`, 13.2% lower latency) at `S=4096`, with a similar result at `S=8192`. This is B200 operator evidence only; no MCore end-to-end or Rubin result exists, so a larger Rubin benefit remains an unmeasured expectation. | Rebuild the production image with v1.28.0 or a pinned post-#684 commit; verify actual dispatch, numerical parity, workspace/HBM, stream behavior, and CUDA Graph composition; then run operator and full-model A/B on GB200, GB300, and Rubin. Do not claim Rubin uplift until measured. | Merged [cuDNN Frontend #684](https://github.com/NVIDIA/cudnn-frontend/pull/684); Rubin architecture enablement [#696](https://github.com/NVIDIA/cudnn-frontend/pull/696).

### MoE execution backends

Priority | Status | Work item | Current finding | Next gate | Tracking
--- | --- | --- | --- | --- | ---
P2 | 📋 | Evaluate MOK for GLM-5.2 MoE | [#6572](https://github.com/NVIDIA/Megatron-LM/pull/6572) has landed a pluggable megakernel backend with Mixture-of-Kittens, replacing post-router dispatch, routed/shared-expert computation, and combine while retaining MCore routing, parameter, optimizer, and checkpoint ownership. Its current evidence covers GB300 proxy and DSv4-Pro, not the 78-layer/256-expert GLM-5.2 stack. GLM is structurally close to the supported envelope, but the current constraints around equal routed/shared intermediate sizes, BF16 shared-expert parameters, the shared-expert output gate, fused gradient accumulation, workspace shape, MoE overlap, recompute, and graph granularity require explicit qualification. | Pin the MCore, MOK, TE, and container revisions; establish BF16/MXFP8 output/loss/gradient, optimizer-step, checkpoint-resume, PP, repeated-MTP, eager, and full-iteration-CG parity; then run same-allocation MOK-versus-HybridEP A/B on GB300 and GB200 at 4K/128K with MTP off/on and both force-balanced and learned routing. Report rank-max latency, tokens/s/GPU, HBM, communication exposure, capacity overflow/fallback, and rank-sampled NSys. | Merged [#6572](https://github.com/NVIDIA/Megatron-LM/pull/6572); [MOK integration branch](https://github.com/QiZhangNV/mixture-of-kittens/tree/qizhang/mcore-integration).

### CPU overhead and CUDA Graph

Priority | Status | Work item | Current finding | Next gate | Tracking
--- | --- | --- | --- | --- | ---
P0 | 🚧 | THD/CP position and reorder construction overhead | Profiling identified `build_packed_allgather_cp_local_positions` as a dominant host-side gap: ordinary DSA rebuilt rank-local packed positions and gathered-KV reorder indices in every layer, including blocking device-to-host work. #6206 now records host cu-seqlens during route prebuild, constructs query positions and a direct inverse reorder from host integers, transfers complete tables asynchronously, and memoizes them per microbatch on `PackedSeqParams`. | Complete review and merge of #6206; require bitwise parity across static/dynamic CP, CP1/CP>1, unequal packs, zero-length entries and padding, then confirm removal of per-layer host synchronization and quantify CPU/GPU NSys and end-to-end gains. | Open [#6206](https://github.com/NVIDIA/Megatron-LM/pull/6206); related generic route primitives [#5664](https://github.com/NVIDIA/Megatron-LM/pull/5664).
P1 | ✅ | Fuse standard RoPE packing and apply indexer RoPE in place | #6343 landed fused standard-RoPE packing for MLA/AbsorbedMLA and the in-place DSA `qk_proj` RoPE path, removing avoidable temporary tensors, memory traffic, and launches. | Retain THD/CP and fused/unfused numerical parity; continue attributing CPU launch time, kernel count, GPU time, and end-to-end impact when the attention path changes. | [#6343](https://github.com/NVIDIA/Megatron-LM/pull/6343).
P1 | ✅ | Whole-MoE paged-stash CUDA Graph foundation | #6022 landed the natural TE PP capture-order integration and paged-stash capture-time microbatch handling without an external correction hook. Broader composition with changing runtime microbatch counts and every GLM graph mode remains a validation concern rather than missing base support. | Retain functional PP, overflow, and dynamic-microbatch coverage; qualify memory/performance in each production graph composition. | [#6022](https://github.com/NVIDIA/Megatron-LM/pull/6022).
P1 | 🚧 | Roll-free MTP future-row alignment | #6741 replaces repeated cumulative full-tensor rolls with immutable-source absolute offsets shared by MTP forward, learned positions, padding, SFT/RL cross-entropy, and TV targets. It supports local CP1, contiguous packed CP, and scheduler-certified one-hop zigzag/dynamic CP; prepares mixed fields with one grouped exchange; and gives fused TV direct source, compact-halo, row-index, and valid-row metadata instead of materializing full-vocabulary target rolls. Unsupported valid layouts retain an atomic compatibility fallback. | Complete stacked review after #6473, retarget to `dev`, and run standard CI plus final signed-head CP1/CP2/CP4 distributed parity. Then close short/long roll-versus-address performance and NSys evidence, CUDA Graph capture/replay, P2P-count invariance across depth, and absence of full-vocabulary target rolls on supported layouts. | Stacked Draft [#6741](https://github.com/NVIDIA/Megatron-LM/pull/6741).
P2 | 🚧 | Graph-compatible DSA | Kernel fusion alone does not close the launch-bound gap: profiling still shows substantial CPU dispatch overhead and GPU bubbles across the composed DSA/AbsorbedMLA self-attention path. #6910, stacked on #6206, removes the dynamic `nonzero`/gather/scatter compaction from ordinary DSA sparse-attention backward and keeps both the combined fused path and the split IndexShare/MTP-share path at fixed row shape. This closes one concrete capture blocker, while #6206 addresses host-side packed position/reorder construction; remaining host-dependent mask/bound construction still needs replay-stable handling. #6206 has since advanced, while #6910 still targets its previous `pull-request/6206` head and is currently conflicting, so the existing #6910 evidence applies to the older stack. | First restack #6910 on the latest #6206 head and rerun focused H100/GB200 parity, capture/replay, and proxy ABBA. Then validate complete DSA/AbsorbedMLA attention capture across changing packed batches and effective CP, requiring stable graph memory and addresses, no stale IndexShare state, reduced CPU launch gaps, and measured end-to-end gain. | Open updated base [#6206](https://github.com/NVIDIA/Megatron-LM/pull/6206); conflicting stacked Draft [#6910](https://github.com/NVIDIA/Megatron-LM/pull/6910).
P2 | 🚧 | Graphable GLM-5.2 | **Partial:** the safe partial strategy is Whole-MoE paged-stash CUDA Graph: capture only the MoE region and leave DSA/AbsorbedMLA attention plus its cross-layer IndexShare and repeated-MTP Index/KV-share state eager. Per-layer attention capture is not the target because the sharing lifetimes cross layer boundaries.
**Full iteration:** one graph can enclose the cross-layer and repeated-MTP reuse dependencies. The aggregate matrix validates a snapshot derived from #5807—not #5807's current source head—with fixed sample lengths, fixed effective CP and metadata per shape, and exactly 32 scheduled microbatches. It improves step time by 12.698%/12.522% for 4K MTP-off/on and 8.024%/10.713% for 128K MTP-off/on, but no-CG always preceded full-CG, so fresh-allocation ABBA is still required for order-independent estimates. It does not validate effective CP or microbatch count changing between iterations. The tested 128K snapshot also retained whole mutable `PackedSeqParams` wrappers in 32 static microbatch slots; transient DSA top-k holders attached to them accounted for 6,144 MiB/rank on PP0–PP2 and 4,096 MiB/rank on PP3. That retention is avoidable snapshot-specific residency, not intrinsic CUDA Graph cost; the current #5807 unpack path creates a fresh wrapper and does not contain this ownership bug.
**Chunk-wise:** this remains the preferred production-dynamic GLM path. Capturing one PP/VPP decoder chunk encloses layer-wise IndexShare, while repeated MTP remains an eager unit, and the chunk graph can replay a dynamic microbatch prefix. Production long-context use still depends on composing chunk capture with the #6801→#6479 dynamic-CP graph-bank stack and keeping every IndexShare group within its captured model-chunk boundary. | Use the landed #6022 foundation for MoE-only partial configurations. First refresh the currently conflicting #5807 onto latest `dev` while preserving fresh-wrapper ownership, add a regression that slot-persistent metadata never retains per-forward DSA holders, and rerun fixed-shape ABBA and all-rank memory. For the preferred production-dynamic path, land and compose #5258, #6206→#6910, and #6801→#6479; narrow the current `dsa_mtp_index_kv_share` validator for chunk capture because MTP stays eager; and validate CP1 plus changing effective CP under THD. Do not treat these open stacks as integrated until graph ownership, slot selection, numerical replay isolation, memory, and performance have been tested together. | Whole-MoE foundation [#6022](https://github.com/NVIDIA/Megatron-LM/pull/6022); conflicting Draft fixed-microbatch source [#5807](https://github.com/NVIDIA/Megatron-LM/pull/5807); Open chunk-wise [#5258](https://github.com/NVIDIA/Megatron-LM/pull/5258); DSA stack [#6206](https://github.com/NVIDIA/Megatron-LM/pull/6206)→[#6910](https://github.com/NVIDIA/Megatron-LM/pull/6910); dynamic-CP stack [#6801](https://github.com/NVIDIA/Megatron-LM/pull/6801)→[#6479](https://github.com/NVIDIA/Megatron-LM/pull/6479).

## Memory

> **Theme status:** Quantized AbsorbedMLA up-projection recompute and per-CP-lane DP all-gather rerouting have landed. Full-model MXFP8 memory is measured on 256×GB300: MTP-off is safe without recompute or activation offload, while the selected MTP7 integration snapshot is safe with selective `[moe_act, layernorm]` recompute and no offload. No controlled no-recompute-versus-selective A/B exists, and the 128K full-CG snapshot retained avoidable DSA top-k state in static microbatch slots. The raw peaks remain valid for that snapshot, but intrinsic graph overhead and the no-recompute prediction must be re-evaluated after fixing ownership. GB200 remains unqualified.

Priority | Status | Work item | Current finding | Next gate | Tracking
--- | --- | --- | --- | --- | ---
P0 | ✅ | FP8/FP4 AbsorbedMLA up-projection recompute | #6178 removed the placeholder quantization assertion after verifying that backward replay restores the recorded quantization recipe and amax context while absorption observes persistent K up-projection weights. GB200 coverage spans tensorwise FP8, MXFP8, and blockwise FP8; SBHD and THD; and combined and split K/V modules, with bitwise eager-versus-recompute output and parameter-gradient parity. | Completed implementation: retain the focused parity coverage and exercise the path in the production GB200 full-model recipe while measuring HBM margin and performance. | [#6178](https://github.com/NVIDIA/Megatron-LM/pull/6178).
P1 | ✅ | Reduce DP-balanced THD rerouting memory | #6378 replaced DP×CP all-to-all with per-CP-lane DP all-gather, avoiding lazily created peer/channel transport buffers. A four-rank GB200 test saved about 1.15 GiB/GPU, while the prior 64-rank transport allocation was estimated at 23.625 GiB/GPU; the tradeoff is higher communication volume. | Validate full-model THD/dynamic-CP with HybridEP on GB200 and GB300; measure device HBM and end-to-end throughput across the logical-CP range. | [#6378](https://github.com/NVIDIA/Megatron-LM/pull/6378).
P1 | 🧪 | Selective recompute and activation offload policy | Quantized `mla_up_proj` recompute is available through #6178. In the aggregate GB300 matrix, MTP-off ran without recompute/offload; MTP7 used selective `[moe_act, layernorm]` recompute and no offload. The calibrated model predicted 276,855 MiB for 128K/MTP7/full-CG without recompute, but that configuration was not run and the prediction predates removal of the static-slot top-k retention described below. The selected recompute configuration measured 261,991 MiB with 22,217 MiB remaining. Current #6472 fails closed when repeated-MTP sharing is combined with any `recompute_granularity`; dependent draft #7107 now owns that composition, so the measurement is historical integration evidence until the current source stack is qualified. | Fix static-slot ownership, recalibrate, and run a controlled same-layout recompute ablation; then qualify the current #6472→#7107 stack, BF16, and GB200. Add only the minimum recompute required by measured HBM; use activation offload only if recompute cannot preserve the safety margin. | Merged foundation [#6178](https://github.com/NVIDIA/Megatron-LM/pull/6178); recompute composition [#7107](https://github.com/NVIDIA/Megatron-LM/pull/7107).
P1 | 🚧 | Full-iteration static-batch ownership for packed DSA | The tested 128K full-CG aggregate snapshot cached whole mutable `PackedSeqParams` wrappers in 32 static microbatch slots. DSA IndexShare attached transient top-k holders to those wrappers, retaining 6,144 MiB/rank on PP0–PP2 and 4,096 MiB/rank on PP3; the observed full-CG allocated-memory deltas were approximately 6,181 and 4,136 MiB respectively. The current #5807 source head creates a fresh wrapper during unpack and is not affected. | Keep only fixed tensors, CP routes, host boundaries, and a namespaced immutable layout cache in static slots; create a fresh lightweight wrapper per unpack; add lifecycle and graph-replay regressions; then rerun the 128K fixed-shape memory matrix before using it for GB200 fit decisions. | Analysis recorded on [#5807](https://github.com/NVIDIA/Megatron-LM/pull/5807#issuecomment-5571226672); source fix/requalification TBD.
P2 | ❓ | Memory-saving fused cuDNN indexer score | cuDNN exposes an additional fused scorer variant that is not wired into Megatron. | Integrate behind the existing backend contract and compare workspace, peak HBM, parity, and kernel time. | TBD.

Latest aggregate GB300 device-memory evidence:

Sequence | MTP | Recompute | No-CG peak / margin (MiB) | Full-CG peak / margin (MiB)
--- | --- | --- | ---: | ---:
4K | Off | None | 225,229 / 58,979 | 225,917 / 58,291
4K | On | `[moe_act, layernorm]` | 240,411 / 43,797 | 240,653 / 43,555
128K | Off | None | 236,351 / 47,857 | 247,509 / 36,699
128K | On | `[moe_act, layernorm]` | 251,847 / 32,361 | 261,991 / 22,217

All eight cells passed the all-rank memory gate against 284,208 MiB of reported GB300 HBM. The two 128K full-CG peaks remain exact but conservative observations of the tested aggregate snapshot: they include avoidable static-slot → `PackedSeqParams` → DSA top-k-holder residency of 6,144 MiB/rank on PP0–PP2 and 4,096 MiB/rank on PP3. Keep the raw measurements, but do not interpret that retained component as intrinsic CUDA Graph overhead or extrapolate it to GB200 fit. The current #5807 source head does not have this ownership bug. GB200 still requires its own full-model calibration.

## Long context and parallelism

> **Theme status:** The underlying THD and CP support is present, but the production workload—ragged samples near 128K packed to 256K with changing logical CP—has not completed correctness, resume, memory, and performance qualification. The 1M-token curriculum remains downstream work and should not start before the 128K/256K path is stable.

Priority | Status | Work item | Current finding | Next gate | Tracking
--- | --- | --- | --- | --- | ---
P1 | 🚧 | THD packed training with dynamic CP | Physical CP is fixed while logical CP/DP groups vary by sample length. Production samples are close to 128K and packed to at most 256K without artificial packed-sequence alignment or a fixed sample count. Every retained DSA path must accept the same packed metadata. | Run short and long ragged batches together with changing logical CP; require deterministic segment boundaries, correct loss masks, no stale metadata, and checkpoint-resume parity. | TBD.
P2 | ❓ | Recompute all-gather CP results | Ordinary DSA currently requires `cp_comm_type=all_gather`: each rank materializes full-sequence gathered K/V and, on indexer-compute layers, gathered indexer K. Retaining these full gathered/reordered activations for backward weakens the activation-memory benefit of CP at long context and large logical CP. | Add a selective policy that retains the rank-local inputs and replay-stable THD/CP metadata, releases eligible full gathered/reordered results after forward, and reconstructs them for backward. Measure saved peak HBM against the extra all-gather/reorder cost; require output and gradient parity across BF16/MXFP8, static/dynamic CP, unequal packs, IndexShare/KVShare, repeated MTP, PP, and checkpoint resume. | TBD.
P2 | 📋 | 1M-token curriculum | Do not begin at full scale. Progress through proxy memory calibration, CP64 preflight, short smoke, and then stability/convergence. | Define sequence-length stages and promotion gates after the 128K/256K path is stable. | TBD.

## Code refactoring

> **Theme status:** None of these cleanups blocks the current DSA baseline. They are deliberately deferred until the functionality and performance contracts stabilize; merging the split/fused DSA paths is the first candidate because it also removes duplicated CP1 indexer and sparse-attention work.

Priority | Status | Work item | Completion rule
--- | --- | --- | ---
P2 | 📋 | Merge DSA split/fused paths | Let the fused path return top-k, prove loss/top-k/gradient parity, support THD+CP1/dynamic CP, then deprecate the split path without aliases.
P2 | 📋 | Unify the remaining KL primitive | DSA and CSA share only genuinely identical loss semantics; native parity remains independent.
P2 | 📋 | Extract a common sparse-attention backend | Move shared kernel dispatch below ordinary DSA and CSA without merging their distinct layouts or architecture contracts.
P2 | 📋 | Split CSA layout conversion into `csa_layout.py` | Keep CSA metadata/layout lowering isolated from DSA-owned APIs.

## Full-model recipes and delivery

> **Theme status:** A paired, pinned-snapshot 256×GB300 MXFP8 matrix establishes runnable full-model layouts and systems baselines for short and long fixed-shape THD workloads. Because every no-CG run preceded its full-CG counterpart, fresh-allocation ABBA is still required for an order-independent graph uplift. This is force-balanced mock-data evidence, not learned-router correctness or convergence qualification. Representative-rank, one-iteration NSys is complete for 128K full-CG; BF16, checkpoint/resume, 4K/no-CG NSys controls, and all GB200 full-model recipes remain open.

Exact source provenance, the complete eight-cell measurements, metric conventions, and validation caveats are recorded in the [GLM-5.2 integration benchmark tracker](https://github.com/NVIDIA/Megatron-LM/pull/6944).

The validated common topology is the 78-layer, 256-expert model on 256 GPUs with `TP1/PP4/EP64/configured-CP64`, dynamic CP, cuDNN DSA, HybridEP, grouped GEMM, MXFP8, `MBS1`, no activation offload, and 32 scheduled microbatches.

MTP mode | Selected pipeline layout | Recompute | Evidence
--- | --- | --- | ---
Off | `Et*18\|t*20\|t*20\|t*20L` | None | Validated 4K and 128K no-CG/full-iteration-CG cells
Repeated MTP7 + E2E-TV + IndexShare/KVShare | `Et*18\|t*24\|t*24\|t*12m*7L` | Selective `[moe_act, layernorm]` | Validated 4K and 128K no-CG/full-iteration-CG cells

The validated aggregate snapshot predates the current PR responsibility split: #6472 now supports repeated-MTP sharing only without activation recompute, while dependent draft #7107 owns recompute composition.

Sequence shape | Valid tokens/sample | Samples/scheduled microbatch | Raw-sample GBS | Effective CP | Effective CP groups | Valid tokens/step
--- | ---: | ---: | ---: | ---: | ---: | ---:
THD 4K | 3,968 | 64 | 2,048 | 1 | 64 | 8,126,464
THD 128K | 126,976 | 2 | 64 | 32 | 2 | 8,126,464

Both shapes preserve 3,968 valid tokens per effective CP rank and the same global-token workload. Dynamic CP was enabled, but every fixed-shape cell used one stable effective-CP layout. Full-iteration CUDA Graph also relied on the fixed 32-microbatch schedule. The MTP-off and MTP-on layouts must therefore remain distinct.

Current recipe qualification:

GPU / precision | Current policy and status
--- | ---
GB300 MXFP8, MTP-off | Validated with no recompute or activation offload for the selected 4K/128K systems workloads.
GB300 MXFP8, MTP7 | Validated with selective `[moe_act, layernorm]` recompute and no activation offload; no controlled recompute ablation yet.
GB300 BF16 | Full-model qualification remains open.
GB200 BF16/MXFP8 | Existing recompute/offload settings remain candidates, not validated delivery recipes; real full-model calibration is required.

NSys status: representative ranks for 128K full-CG have complete one-iteration traces for both MTP-off and MTP-on. The 4K cells, no-CG controls, and GB200 traces remain open; broader PP-stage/rank coverage should be added only where the existing sample cannot answer a concrete imbalance or communication question.

For each delivery recipe:

- Place each PP stage and its EP64 group within one NVL domain; verify the actual rank/node/domain mapping rather than relying only on `segment=16`.
- Record resolved arguments, source/container SHA, topology, median/p95 step time, tokens/s/GPU, corrected TFLOPS/GPU when available, and peak device HBM.
- Run unprofiled performance separately from rank-sampled, one-iteration NSys capture; do not compare profiled iteration time against the performance result.
- Validate loss, indexer loss, repeated-MTP loss, gradients, parameter norms, checkpoint resume, and absence of overflow or fallback on the full model.
- Record whether force balancing is enabled, and never use a force-balanced result as correctness, convergence, SFT, or RL evidence.
- Treat GB200 and GB300 as separate recipe results because their recompute/offload policies may differ.