NVIDIA / NVIDIA/Megatron-LM

CI perf regression: gpt_583m_perf_gb200_4gpu on GB200

Open
#5,692 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 3h
Merged PRs (30d)
272

Description

**Describe the bug**

Performance test `gpt_583m_perf_gb200_4gpu` failed the GB200 perf baseline threshold while validating PR [#5632: Update base image to nvcr.io/nvidia/pytorch:26.06-py3](https://github.com/NVIDIA/Megatron-LM/pull/5632).

**Failing metrics**

| Batch | Metric | Measured | Baseline | Delta | Threshold |
| --- | ---: | ---: | ---: | ---: | ---: |
| `8` | `throughput_tok_per_sec` | `242.994` | `298.886` | `-18.70%` | floor `268.998` |
| `8` | `tpot_ms_per_tok` | `32.923` | `26.766` | `+23.00%` | ceiling `29.443` |
| `128` | `throughput_tok_per_sec` | `3773.043` | `4663.364` | `-19.09%` | floor `4197.028` |
| `128` | `avg_latency_ms` | `3838.628` | `3324.592` | `+15.46%` | ceiling `3657.051` |
| `128` | `tpot_ms_per_tok` | `33.925` | `27.448` | `+23.60%` | ceiling `30.193` |

`batch_32` throughput was also lower than baseline (`1130.042` vs `1188.752`, `-4.94%`) but remained within tolerance.

**Additional context**

Tracking issue created while marking this test `allow_failure` so PR #5632 can continue while the regression is investigated.

Contributor guide

Open the contributing guide

Research direction

Start with the gpt_583m_perf_gb200_4gpu performance test and compare its GB200 results against the baselines reported here. Review PR #5632 and its nvcr.io/nvidia/pytorch:26.06-py3 base-image change, then investigate the failing throughput and latency metrics. Done means the regression is explained and the test no longer requires allow_failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
performance, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.