🐛 Umbrella: CI failures in PR #4992 Transformer Engine 2.16.0 update
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 272
Description
**Describe the bug**
Umbrella tracker for two CI failures observed while validating [PR #4992: chore: Update transformer-engine dependency to version 2.16.0](https://github.com/NVIDIA/Megatron-LM/pull/4992).
Tag the [@mcore-oncall](https://github.com/orgs/NVIDIA/teams/mcore-oncall) to get oncall's attention to this issue.
**Child issues**
- [ ] #5025
- [ ] #5026
**Failing runs**
| Child | Failure | Run | Job |
|-------|---------|-----|-----|
| #5025 | `tests/unit_tests/transformer/moe/test_aux_loss.py::TestRouterAuxLoss::test_aux_loss_fusion_equivalence[seq_aux_loss]` | [26467462352](https://github.com/NVIDIA/Megatron-LM/actions/runs/26467462352) | [tests/unit_tests/transformer/moe/**/*.py - latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/26467462352/job/77944314720) |
| #5026 | `moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node` | [26476200406](https://github.com/NVIDIA/Megatron-LM/actions/runs/26476200406) | [moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node - latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/26476200406/job/77969503276) |
**Error summary**
```text
#5025:
RuntimeError: fused_moe_aux_loss_forward_kernel_launcher: Assertion failed: num_experts == num_cols.
Number of experts (32) must be equal to number of input columns (64).
#5026:
ERROR:megatron.core.rerun_state_machine:Unexpected result nan at iteration #3
pretrain_gpt.py FAILED
##[error]gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node — FAILED (exit 1)
```
**Additional context**
Triaged automatically via `/triage-issue`.
Contributor guide
Assessment
This issue has not been assessed yet.