CI perf regression: gpt_583m_perf_gb200_4gpu on GB200
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 272
Description
**Describe the bug**
Performance test `gpt_583m_perf_gb200_4gpu` failed the GB200 perf baseline threshold while validating PR [#5632: Update base image to nvcr.io/nvidia/pytorch:26.06-py3](https://github.com/NVIDIA/Megatron-LM/pull/5632).
**Failing metrics**
| Batch | Metric | Measured | Baseline | Delta | Threshold |
| --- | ---: | ---: | ---: | ---: | ---: |
| `8` | `throughput_tok_per_sec` | `242.994` | `298.886` | `-18.70%` | floor `268.998` |
| `8` | `tpot_ms_per_tok` | `32.923` | `26.766` | `+23.00%` | ceiling `29.443` |
| `128` | `throughput_tok_per_sec` | `3773.043` | `4663.364` | `-19.09%` | floor `4197.028` |
| `128` | `avg_latency_ms` | `3838.628` | `3324.592` | `+15.46%` | ceiling `3657.051` |
| `128` | `tpot_ms_per_tok` | `33.925` | `27.448` | `+23.60%` | ceiling `30.193` |
`batch_32` throughput was also lower than baseline (`1130.042` vs `1188.752`, `-4.94%`) but remained within tolerance.
**Additional context**
Tracking issue created while marking this test `allow_failure` so PR #5632 can continue while the regression is investigated.
Contributor guide
Research direction
Start with the gpt_583m_perf_gb200_4gpu performance test and compare its GB200 results against the baselines reported here. Review PR #5632 and its nvcr.io/nvidia/pytorch:26.06-py3 base-image change, then investigate the failing throughput and latency metrics. Done means the regression is explained and the test no longer requires allow_failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- performance, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100