CI perf regression: hybrid_2b_perf batch_32 on GB200
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
**Describe the bug**
Performance test `hybrid_2b_perf` failed the GB200 perf baseline threshold while validating PR [#5632: Update base image to nvcr.io/nvidia/pytorch:26.06-py3](https://github.com/NVIDIA/Megatron-LM/pull/5632).
**Failing metrics**
| Batch | Metric | Measured | Baseline | Delta | Threshold |
| --- | ---: | ---: | ---: | ---: | ---: |
| `32` | `throughput_tok_per_sec` | `961.963` | `1081.357` | `-11.04%` | floor `973.221` |
| `32` | `avg_latency_ms` | `4253.818` | `3783.034` | `+12.44%` | ceiling `4161.337` |
| `32` | `p50_latency_ms` | `4238.443` | `3807.536` | `+11.32%` | ceiling `4188.290` |
| `32` | `tpot_ms_per_tok` | `33.265` | `29.592` | `+12.41%` | ceiling `32.552` |
Other batches passed their thresholds.
**Additional context**
Tracking issue created while marking this test `allow_failure` so PR #5632 can continue while the regression is investigated.
Contributor guide
Research direction
Start with the `hybrid_2b_perf` performance test and the validation for PR #5632's base-image update. Reproduce the batch-32 run on GB200, compare its throughput and latency metrics with the listed baseline, and trace the regression before changing anything. Done means the regression is explained and the batch-32 metrics meet their thresholds, or the tracking issue records a concrete cause and follow-up.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, performance, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100