NVIDIA / NVIDIA/Megatron-LM

🐛 CI failure: determinism_perf exceeds 1.25x slowdown threshold

Open
#5,747 1 comment 0 reactions 1 assignee Claimed by @balasaajay View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Describe the bug**

CI performance test `gpt/determinism_perf` failed in job [`gpt/determinism_perf - latest`](https://github.com/NVIDIA/Megatron-LM/actions/runs/29105388368/job/86417629996). At steady-state iteration 5, deterministic mode was 1.30x slower than non-deterministic mode, exceeding the configured 1.25x threshold.

Tag @NVIDIA/mcore-oncall to get oncall attention to this issue.

**Failing run**

| Field | Value |
|-------|-------|
| PR | [#5680: build: Update Transformer Engine to 2.17](https://github.com/NVIDIA/Megatron-LM/pull/5680) |
| Run | [29105388368](https://github.com/NVIDIA/Megatron-LM/actions/runs/29105388368) |
| Job | [gpt/determinism_perf - latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/29105388368/job/86417629996) |

**Error**

```text
step_time iter=5: det=72.40ms nondet=55.50ms ratio=1.30x (threshold 1.25x)
FAIL: det 1.30x slower than nondet (> 1.25x)
```

**Steps/Code to reproduce bug**

Re-run the failing CI job linked above, or locally on an 8-GPU H100 node inside the dev container:

```bash
bash tests/performance_tests/shell_test_utils/determinism/perf_breakdown.sh \
/tmp/perf-leaderboards \
/tmp/determinism-perf-logs
```

**Additional context**

- Triggering commit: `bccae74c1bad728b9ecf806bd5005d70260cd8cc`.
- This was observed on PR #5680, which updates Transformer Engine and container dependencies; the single failing measurement does not by itself establish causality.
- The nearby `torch.distributed.elastic` SIGTERM traceback occurs when Nsight Systems ends capture with `--capture-range-end=stop-shutdown`; the explicit test failure is the performance-ratio check above.
- An `AccumulateGrad` stream-mismatch warning was also emitted, but it did not stop training.
- Full logs and uploaded profiling artifacts are available from the linked job.
- Triaged automatically via `/triage-issue`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.