NVIDIA / NVIDIA/Megatron-LM

Weekly CI failure: GB200 Nemotron3 Super mem-max-allocated mismatch

Open
#5,457 0 comments 0 reactions 1 assignee Claimed by @balasaajay View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Describe the bug**

The GB200 weekly release test `nemotron3_super_release_gb200_sm` is failing on the latest run because the `mem-max-allocated-bytes` metric is outside the 5% approximate comparison tolerance.

Tag @NVIDIA/mcore-oncall to get oncall's attention to this issue.

**Failing run**

| Field | Value |
|-------|-------|
| Parent scheduled pipeline | `55286509` |
| GB200 child pipeline | `55288455` |
| Commit | `d1410e15` - Add MIMO runtime setup: per-role RNG seeding and DDP wrapping (#5285) |
| Latest failing job | `344197346` |
| Job name | `nemotron3_super_release_gb200_sm` |
| Test case | `tests/functional_tests/test_cases/nemotron/nemotron3_super_release_gb200_sm` |

**History**

| Date | Pipeline / job | Status | Note |
|------|----------------|--------|------|
| 2026-05-15 | first scheduled run | failed | Earliest scheduled run observed for this GB200-sm case; run logs use `weekly-2026-05-16` |
| 2026-06-20 | `55288455` / `344197346` | failed | Latest failure, current signature is `mem-max-allocated-bytes` only |

Earlier sampled runs for this test had related but different signatures, including a golden/actual shape mismatch and `lm loss` failures. This issue is scoped to the latest `mem-max-allocated-bytes` mismatch.

**Error**

```text
tests/functional_tests/python_test_utils/test_pretraining_regular_pipeline.py::test_regular_pipeline

APPROXIMATE test for metric iteration-time: PASSED
APPROXIMATE test for metric lm loss: PASSED
APPROXIMATE test for metric mem-allocated-bytes: PASSED
Approximate comparison of mem-max-allocated-bytes: FAILED

FAILED tests/functional_tests/python_test_utils/test_pretraining_regular_pipeline.py::test_regular_pipeline - AssertionError: The following metrics failed: mem-max-allocated-bytes
```

Deviation summary for the latest failing job:

| Metric | Tolerance | Failed points | Max deviation | Median failing deviation | Example actual vs golden |
|--------|-----------|---------------|---------------|--------------------------|--------------------------|
| `mem-max-allocated-bytes` | 5% | 11 / 954, allowed 9 | 11.73% | 11.65% | 110.34B vs 125.01B, lower by 13.66 GiB |

All other checked metrics passed in the latest failure.

**Steps/Code to reproduce bug**

Run the GB200 release functional test for:

```bash
pytest tests/functional_tests/python_test_utils/test_pretraining_regular_pipeline.py::test_regular_pipeline
```

using:

```text
TRAINING_PARAMS_PATH=./tests/functional_tests/test_cases/nemotron/nemotron3_super_release_gb200_sm/model_config.yaml
GOLDEN_VALUES_PATH=./tests/functional_tests/test_cases/nemotron/nemotron3_super_release_gb200_sm/golden_values_dev_dgx_gb200.json
```

**Additional context**

- Duplicate searches for `nemotron3_super_release_gb200_sm` and `Nemotron GB200 memory mismatch mem-max-allocated` did not find an open matching issue.
- CI system links are intentionally omitted; pipeline and job IDs are included for internal lookup.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.