NVIDIA / NVIDIA/Megatron-LM

[dev] 26.04 Bump breakages

Open
#5,053 0 comments 0 reactions 1 assignee Claimed by @FDecaYed View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 3h
Merged PRs (30d)
272

Description

**Describe the bug**

Umbrella issue tracking CI breakages on the `dev` branch surfaced by bumping the base container image from `nvcr.io/nvidia/pytorch:26.02-py3` to `nvcr.io/nvidia/pytorch:26.04-py3` (PR #5051).

The bump exposes pre-existing skew between the GitHub-CI pin (`docker/.ngc_version.dev`) and the GitLab-CI pin (`.gitlab/stages/01.build.yml` → `CI_MCORE_DEV_IMAGE`). On `dev`, the GitHub pin was at `26.02-py3` while the GitLab pin was already at `26.04-py3`, so the two lanes produced bit-different numerics for deterministic functional tests. PR #5051 syncs GitHub to `26.04-py3` and is now flushing out the latent regressions on the 26.04 container.

Tag @NVIDIA/mcore-oncall.

**Failing run**

| Field | Value |
|-------|-------|
| PR | [#5051: chore: Update Docker image version to 26.04-py3 on dev](https://github.com/NVIDIA/Megatron-LM/pull/5051) |
| Run | [26627543568](https://github.com/NVIDIA/Megatron-LM/actions/runs/26627543568) |
| Base | `dev` |
| SHA | `08d136568361b9347eee916c04c0bbabfae6af30` |

**Tracked sub-issues** (per error type)

- [ ] #5054 — Tolerance failure: `TestFusedDSAIndexerLossGradient::test_fused_indexer_loss_gradient_matches_autograd[True]` (grad_q max diff `8.59e-05`)
- [ ] #5055 — Tile compiler crash: all fused mHC kernel tests in `tests/unit_tests/fusions/test_fused_mhc_kernels.py` (`cuda.tile._exception.TileCompilerExecutionError` for `sm_90`) — `TestFusedSinkhorn`, `TestFusedHAggregate`, `TestFusedHPostBDA`, `TestFusedProjRms`, `TestEndToEndFused`
- [ ] #5056 — MBridge downstream container build: `uv sync` network timeout pulling `nvidia-cusparse==12.6.3.3`
- [ ] #5059 — Golden-value drift (GitHub `mr-github`): 72 functional tests fail deterministic compare on 26.04 (refresh via `download_golden_values.py --source github --pipeline-id 26631592359 --only-failing`)
- [ ] #5061 — Golden-value drift (GitLab `L2`/nightly): 61 functional tests on dev_dgx_{a100,h100,gb200} fail deterministic compare on 26.04 (refresh via `download_golden_values.py --source gitlab --pipeline-id 53056319 --only-failing`)

**Steps/Code to reproduce bug**

Re-run [the failing CI workflow](https://github.com/NVIDIA/Megatron-LM/actions/runs/26627543568) against `dev` at the synced 26.04 pin, or locally inside the new dev container:

```bash
echo 'nvcr.io/nvidia/pytorch:26.04-py3' > docker/.ngc_version.dev
# (GitLab pin .gitlab/stages/01.build.yml is already at 26.04)
docker build --target main -f docker/Dockerfile.ci.dev \
--build-arg FROM_IMAGE_NAME=$(cat docker/.ngc_version.dev) .
```

**Additional context**

Triaged automatically via the `create-issue` skill. Each linked sub-issue captures one distinct error type; this umbrella issue is for coordination and merge-gate visibility.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.