Determinism roadmap tracking

This issue tracks the Megatron-Core determinism effort. User-facing documentation describes the supported contract; this issue records current implementation and validation status.

## Contract

Determinism claims are split into three gates:

1. **Fixed-environment repeatability:** two runs with the same model, data, seed, software, hardware class, and topology produce bit-exact training metrics and semantic trace events.
2. **Baseline equivalence:** a performance optimization may be repeatable but must also be compared with the established deterministic baseline. A baseline-changing optimization needs separate convergence qualification.
3. **Restart replay:** a resumed run must reproduce the uninterrupted run from the checkpoint boundary.

The production goal is bit-exact fixed-environment training with less than 10% steady-state step-time overhead; approximately 5% remains the stretch target.

## Landed

- #5041 — consolidated `--deterministic-mode` validation and environment defaults; module-level bit-exact coverage across GPT, Transformer Block, and Hybrid Model configurations; and a deterministic-vs-nondeterministic profiling gate.

## In review

- #5718 — developer documentation: contract, training-path surface area, operator catalog, validation evidence, and glossary.
- #5729 — DeepSeek-V4/DSA module-level deterministic proxy.

## Verified findings

- DeepSeek-V3-style and DeepSeek-V4-style 32-GPU proxy configurations repeat bit-exactly.
- Nemotron-3 Ultra has repeated bit-exactly at 48 nodes for 50 iterations in completed paired runs. Semantic traces also matched without recompute, collective, Transformer Engine backend, or optimizer-boundary divergence. A fresh repeat in the original launch environment is in progress before the scale gate is closed.
- Disabling `torch.utils.deterministic.fill_uninitialized_memory` remained baseline-bit-exact for 20-iteration DeepSeek-V3-style and Nemotron-3 Ultra runs and improved steady-state performance by approximately 16% and 18%, respectively.
- `--moe-router-fusion` is independently repeatable, but it changes the established deterministic baseline from the first few iterations. It is therefore not currently accepted as a baseline-preserving deterministic optimization; convergence qualification is required before production adoption.
- The primary Nemotron-3 Ultra defects found so far were topology-dependent final gradient reductions and external Mamba memory-efficient Triton autotuning. Candidate fixes exist on the working branch and still require scoped upstream review.
- The Nemotron-4 debug configuration is repeatable, but the production recipe is not yet qualified. Remaining blockers include a deterministic fast FLA/GDN path, an upstream cuDNN dropout-probability issue, and the SM100 head-dimension-256 kernel limitation.

## Active workstreams and gates

1. **Scale effectiveness** — finish the independent 48-node Nemotron-3 Ultra repeat, then run the prepared 3,072-GPU qualification only with an assigned reservation. Passing smaller runs is evidence, not proof, for the 3,072-GPU topology.
2. **Scale-safe localization** — use hierarchical tracing: low-cost iteration/phase/collective summaries on all ranks; targeted rank groups and device-side digests after a mismatch; full-byte or full-op tracing only around the first divergent boundary. Sampled/device-digest traces are diagnostic matches, not cryptographic certificates.
3. **Model coverage** — upstream scoped correctness tests and recurring deterministic-vs-nondeterministic performance coverage for DeepSeek-V3, DeepSeek-V4, and Nemotron hybrid proxies.
4. **Performance** — report steady-state windows and paired orderings, excluding initialization, first-iteration compilation, and profiler export. Treat less than 1% overhead as acceptable, more than 7% as a blocker, and 1–7% as an explicit product decision.
5. **Nemotron-4 enablement** — close the FLA/GDN, cuDNN dropout, and SM100 head-dimension blockers independently; do not describe debug-kernel repeatability as production-recipe qualification.

## Immediate next actions

- Close the fresh 48-node Nemotron-3 Ultra A/A certificate.
- Submit the topology-independent gradient-reduction fix, Mamba autotuner pinning, and hierarchical tracer as independently reviewable changes with their tests.
- Add scheduled model-level correctness and performance runs.
- Keep the immutable 3,072-GPU qualification bundle ready, but do not launch without a reservation.

Updates will be posted here as these gates close.

Start by reviewing #5041, #5718, and #5729, then inspect the current deterministic validation and model-level coverage described in the workstreams. This roadmap is complete only when the listed scale, localization, model-coverage, performance, and Nemotron-4 qualification gates are independently closed and documented.

Written by the indexing model from the issue text.