🐛 CI failure: determinism_perf exceeds 1.25x slowdown threshold
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
**Describe the bug**
CI performance test `gpt/determinism_perf` failed in job [`gpt/determinism_perf - latest`](https://github.com/NVIDIA/Megatron-LM/actions/runs/29105388368/job/86417629996). At steady-state iteration 5, deterministic mode was 1.30x slower than non-deterministic mode, exceeding the configured 1.25x threshold.
Tag @NVIDIA/mcore-oncall to get oncall attention to this issue.
**Failing run**
| Field | Value |
|-------|-------|
| PR | [#5680: build: Update Transformer Engine to 2.17](https://github.com/NVIDIA/Megatron-LM/pull/5680) |
| Run | [29105388368](https://github.com/NVIDIA/Megatron-LM/actions/runs/29105388368) |
| Job | [gpt/determinism_perf - latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/29105388368/job/86417629996) |
**Error**
```text
step_time iter=5: det=72.40ms nondet=55.50ms ratio=1.30x (threshold 1.25x)
FAIL: det 1.30x slower than nondet (> 1.25x)
```
**Steps/Code to reproduce bug**
Re-run the failing CI job linked above, or locally on an 8-GPU H100 node inside the dev container:
```bash
bash tests/performance_tests/shell_test_utils/determinism/perf_breakdown.sh \
/tmp/perf-leaderboards \
/tmp/determinism-perf-logs
```
**Additional context**
- Triggering commit: `bccae74c1bad728b9ecf806bd5005d70260cd8cc`.
- This was observed on PR #5680, which updates Transformer Engine and container dependencies; the single failing measurement does not by itself establish causality.
- The nearby `torch.distributed.elastic` SIGTERM traceback occurs when Nsight Systems ends capture with `--capture-range-end=stop-shutdown`; the explicit test failure is the performance-ratio check above.
- An `AccumulateGrad` stream-mismatch warning was also emitted, but it did not stop training.
- Full logs and uploaded profiling artifacts are available from the linked job.
- Triaged automatically via `/triage-issue`.
Contributor guide
Assessment
This issue has not been assessed yet.