NVIDIA / NVIDIA/Megatron-LM

🐛 CI failure: all fused mHC kernel tests — cuda-tile compiler crash on 26.04 (sm_90)

Open
#5,055 0 comments 0 reactions 1 assignee Claimed by @jingqiny-99 View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 3h
Merged PRs (30d)
272

Description

**Describe the bug**

Every `_require_cutile`-decorated test class in `tests/unit_tests/fusions/test_fused_mhc_kernels.py` fails on the 26.04 base image with the same `cuda.tile._exception.TileCompilerExecutionError: Return code 1` from `/usr/local/cuda/bin/tileiras`. The 26.02 container did not exhibit this. It is a toolchain regression triggered by the bump, not a code regression on `dev`.

Affected classes (all in `tests/unit_tests/fusions/test_fused_mhc_kernels.py`):

- `TestFusedSinkhorn::test_fwd_bwd_vs_reference` — surfaced in run `26627543568` (job [78473355987](https://github.com/NVIDIA/Megatron-LM/actions/runs/26627543568/job/78473355987))
- `TestFusedHAggregate::test_fwd_bwd_vs_reference[2-4-4-1024]` — surfaced in run `26630681301` (job [78478726837](https://github.com/NVIDIA/Megatron-LM/actions/runs/26630681301/job/78478726837)) after `TestFusedSinkhorn` was quarantined
- `TestFusedHPostBDA::test_fwd_bwd_vs_reference` — pre-emptively quarantined (same toolchain crash expected; `pytest-random-order` would expose it next)
- `TestFusedProjRms::test_fwd_bwd_vs_reference` — pre-emptively quarantined (same)
- `TestEndToEndFused::test_full_pipeline_fwd_bwd` — pre-emptively quarantined (composes all four fused kernels)

All five are now marked `@pytest.mark.flaky_in_dev` on PR #5051 so the dev pipeline can stay green while the upstream toolchain is investigated.

Tracked under umbrella issue #5053. Tag @NVIDIA/mcore-oncall.

**Failing runs**

| Field | Value |
|-------|-------|
| PR | [#5051: chore: Update Docker image version to 26.04-py3 on dev](https://github.com/NVIDIA/Megatron-LM/pull/5051) |
| First run (Sinkhorn) | [26627543568 — job 78473355987](https://github.com/NVIDIA/Megatron-LM/actions/runs/26627543568/job/78473355987) |
| Follow-up run (HAggregate) | [26630681301 — job 78478726837](https://github.com/NVIDIA/Megatron-LM/actions/runs/26630681301/job/78478726837) |
| Quarantine commit | [`795f87be50`](https://github.com/NVIDIA/Megatron-LM/pull/5051/commits/795f87be50fe2410c5569d3000b46f6f09a6702d) |

**Error (representative)**

```
E subprocess.CalledProcessError: Command '['/usr/local/cuda/bin/tileiras',
'/tmp/tmp.../.bytecode',
'-o', '/tmp/tmp.../.cubin',
'--gpu-name', 'sm_90', '-O3', '--lineinfo']' returned non-zero exit status 1.
E cuda.tile._exception.TileCompilerExecutionError: Return code 1
```

**Steps/Code to reproduce bug**

Re-run any of the failing jobs above, or locally inside the 26.04 dev container:

```bash
# Hits every affected class:
pytest tests/unit_tests/fusions/test_fused_mhc_kernels.py -m "not flaky_in_dev" --co # to list
pytest tests/unit_tests/fusions/test_fused_mhc_kernels.py::TestFusedHAggregate
```

**Additional context**

Triaged automatically via the `create-issue` skill; updated when `TestFusedHAggregate` surfaced as a sibling failure after `TestFusedSinkhorn` was quarantined (`pytest-random-order` reshuffles which class hits the broken toolchain first, so per-class whack-a-mole is ineffective — quarantining all five at once stops it).

Action items:

- Capture the `tileiras` stderr from the runner artifacts.
- File a toolchain bug against the 26.04 NGC PyTorch image (`nvcr.io/nvidia/pytorch:26.04-py3`) if reproducible standalone.
- Once fixed upstream (or workaround landed), remove the `flaky_in_dev` markers from the five classes in one commit.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.