NVIDIA / NVIDIA/Megatron-LM

🐛 Umbrella: CI failures in PR #4992 Transformer Engine 2.16.0 update

Open
#5,027 1 comment 0 reactions 1 assignee Claimed by @FDecaYed View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 3h
Merged PRs (30d)
272

Description

**Describe the bug**

Umbrella tracker for two CI failures observed while validating [PR #4992: chore: Update transformer-engine dependency to version 2.16.0](https://github.com/NVIDIA/Megatron-LM/pull/4992).
Tag the [@mcore-oncall](https://github.com/orgs/NVIDIA/teams/mcore-oncall) to get oncall's attention to this issue.

**Child issues**

- [ ] #5025
- [ ] #5026

**Failing runs**

| Child | Failure | Run | Job |
|-------|---------|-----|-----|
| #5025 | `tests/unit_tests/transformer/moe/test_aux_loss.py::TestRouterAuxLoss::test_aux_loss_fusion_equivalence[seq_aux_loss]` | [26467462352](https://github.com/NVIDIA/Megatron-LM/actions/runs/26467462352) | [tests/unit_tests/transformer/moe/**/*.py - latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/26467462352/job/77944314720) |
| #5026 | `moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node` | [26476200406](https://github.com/NVIDIA/Megatron-LM/actions/runs/26476200406) | [moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node - latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/26476200406/job/77969503276) |

**Error summary**

```text
#5025:
RuntimeError: fused_moe_aux_loss_forward_kernel_launcher: Assertion failed: num_experts == num_cols.
Number of experts (32) must be equal to number of input columns (64).

#5026:
ERROR:megatron.core.rerun_state_machine:Unexpected result nan at iteration #3
pretrain_gpt.py FAILED
##[error]gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node — FAILED (exit 1)
```

**Additional context**

Triaged automatically via `/triage-issue`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.