NVIDIA-NeMo / NVIDIA-NeMo/Automodel
[Tracking] E2E training alignment and convergence against Megatron/Megatron-Bridge
@HuiyingLi is already working on this.
Since Jul 11, 2026.
- Dominant language
- Python
- Stars
- 963
- Forks
- 318
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 143
Description
We should add an end-to-end training alignment test suite that compares AutoModel against Megatron/Megatron-Bridge under identical settings. The goal is to catch subtle correctness issues in the full training path, including tokenizer/chat rendering, sample packing, answer-only loss masks, reasoning-content masks, position ids, forward numerics, backward numerics, and optimizer-step behavior.
Recent answer-only loss-mask issues in multiturn chat rendering show that component-level tests are not enough. Some bugs may not reliably surface as obvious training failures, but can still change the effective objective or training dynamics. We need a reference-based alignment workflow for representative model families.
Scope
Compare AutoModel training against Megatron/Megatron-Bridge for these model families:
- Qwen3
- Qwen3.5
- Gemma4
- DeepSeek v4
- GLM 5.2
The test should cover the full training process, not only chat template rendering.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.