NVIDIA-NeMo / NVIDIA-NeMo/Megatron-Bridge
[bug] [Determinism] determinism break of ultra v3 on large scale
Open
@Connor-XY is already working on this.
Since Jul 16, 2026.
area:model
bug
Determinism
tracking
- Dominant language
- Python
- Stars
- 920
- Forks
- 505
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 253
Description
Problem
24 nodes (96 GPUs) -> 768 nodes (3072 gpus)
Non-deterministic scaling: bit-exact ✓ -> diverges from iteration 1 ✗
Need to narrow down the root cause of determinism failure.
Minimal repro
follow: https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/zhiyul/nemotron-3-ultra-perf-recipe
Expected behavior
Determinism stands in large scale like 3k gpu run.
Affected area
area:model
Regression?
Yes
Environment
No response
Logs
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.