NVIDIA / NVIDIA/Megatron-LM

CI perf regression: hybrid_2b_perf batch_32 on GB200

Open
#5,693 4 comments 0 reactions 0 assignees View on GitHub
bug ci
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Describe the bug**

Performance test `hybrid_2b_perf` failed the GB200 perf baseline threshold while validating PR [#5632: Update base image to nvcr.io/nvidia/pytorch:26.06-py3](https://github.com/NVIDIA/Megatron-LM/pull/5632).

**Failing metrics**

| Batch | Metric | Measured | Baseline | Delta | Threshold |
| --- | ---: | ---: | ---: | ---: | ---: |
| `32` | `throughput_tok_per_sec` | `961.963` | `1081.357` | `-11.04%` | floor `973.221` |
| `32` | `avg_latency_ms` | `4253.818` | `3783.034` | `+12.44%` | ceiling `4161.337` |
| `32` | `p50_latency_ms` | `4238.443` | `3807.536` | `+11.32%` | ceiling `4188.290` |
| `32` | `tpot_ms_per_tok` | `33.265` | `29.592` | `+12.41%` | ceiling `32.552` |

Other batches passed their thresholds.

**Additional context**

Tracking issue created while marking this test `allow_failure` so PR #5632 can continue while the regression is investigated.

Contributor guide

Open the contributing guide

Research direction

Start with the `hybrid_2b_perf` performance test and the validation for PR #5632's base-image update. Reproduce the batch-32 run on GB200, compare its throughput and latency metrics with the listed baseline, and trace the regression before changing anything. Done means the regression is explained and the batch-32 metrics meet their thresholds, or the tracking issue records a concrete cause and follow-up.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.