🐛 CI failure: moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node

**Describe the bug**

CI functional test `moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node` failed in job [`moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node - latest`](https://github.com/NVIDIA/Megatron-LM/actions/runs/26476200406/job/77969503276).
Tag the [@mcore-oncall](https://github.com/orgs/NVIDIA/teams/mcore-oncall) to get oncall's attention to this issue.

**Failing run**

| Field | Value |
|-------|-------|
| PR | [#4992: chore: Update transformer-engine dependency to version 2.16.0](https://github.com/NVIDIA/Megatron-LM/pull/4992) |
| Run | [26476200406](https://github.com/NVIDIA/Megatron-LM/actions/runs/26476200406) |
| Job | [moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node - latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/26476200406/job/77969503276) |

**Error**

```text
ERROR:megatron.core.rerun_state_machine:Unexpected result nan on rank 3 at iteration #3 invocation #2 (message='found NaN in local grad norm for bucket #0 in backward pass before data-parallel communication collective')
ERROR:megatron.core.rerun_state_machine:Rank 3, node core-ci-test-1779833921-task-1-0, device 3, iteration #3: First rerun: unexpected result is reproducible within the tolerance (nan = nan). Need to rerun on a different GPU to verify correctness.!
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
pretrain_gpt.py FAILED

ERROR:megatron.core.rerun_state_machine:Unexpected result nan on rank 3 at iteration #3 invocation #1 (message='found NaN in local forward loss calculation')
ERROR:megatron.core.rerun_state_machine:Unexpected result nan on rank 0 at iteration #3 invocation #1 (message='found NaN in local forward loss calculation')
ERROR:megatron.core.rerun_state_machine:Rank 3, node core-ci-test-1779833989-task-1-0, device 3, iteration #3: First rerun: unexpected result is reproducible within the tolerance (nan = nan). Need to rerun on a different GPU to verify correctness.!
ERROR:megatron.core.rerun_state_machine:Rank 0, node core-ci-test-1779833989-task-1-0, device 0, iteration #3: First rerun: unexpected result is reproducible within the tolerance (nan = nan). Need to rerun on a different GPU to verify correctness.!
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
pretrain_gpt.py FAILED

##[error]gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node — FAILED (exit 1)
```

**Steps/Code to reproduce bug**

Re-run the failing CI job linked above, or run the functional test case from the dev container:

```bash
tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node
```

**Additional context**

Functional test case files:

- `tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node/model_config.yaml`
- `tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node/golden_values_dev_dgx_gb200.json`

Umbrella issue: #5027.

Triaged automatically via `/triage-issue`.