🐛 CI failure: test_paged_stashing.py::TestPagedStashing::test_forward_backward_4_layers
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 272
Description
**Describe the bug**
CI test `tests/unit_tests/transformer/moe/test_paged_stashing.py::TestPagedStashing::test_forward_backward_4_layers` failed in job [tests/unit_tests/transformer/moe/**/*.py - latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/24510353170/job/71641206557).
Tag the [@mcore-oncall](https://github.com/orgs/NVIDIA/teams/mcore-oncall) to get oncall's attention to this issue.
**Failing run**
| Field | Value |
|-------|-------|
| PR | [#4332: build: bump TransformerEngine to release_v2.14 (dev)](https://github.com/NVIDIA/Megatron-LM/pull/4332) |
| Run | [24510353170](https://github.com/NVIDIA/Megatron-LM/actions/runs/24510353170) |
| Job | [71641206557](https://github.com/NVIDIA/Megatron-LM/actions/runs/24510353170/job/71641206557) |
**Error**
```
tests/unit_tests/transformer/moe/test_paged_stashing.py::TestPagedStashing::test_forward_backward_4_layers
File "megatron/core/transformer/moe/experts.py", line 730, in TEGroupedMLP.__init__
if self.config.use_transformer_engine_op_fuser:
assert (
> self._is_fused_impl_supported()
), "Fused GroupedMLP is not supported for this configuration."
E AssertionError: Fused GroupedMLP is not supported for this configuration.
megatron/core/transformer/moe/experts.py:730: AssertionError
```
The test constructs a `MoEModelTestContainer` with `use_transformer_engine_op_fuser=True` and `moe_paged_stash=True`. The `TEGroupedMLP.__init__` guard at `experts.py:730` rejects this combination because `_is_fused_impl_supported()` returns `False`, indicating the fused TE op fuser path does not support paged stashing.
**Steps/Code to reproduce bug**
Re-run the failing CI job linked above, or locally inside the dev container:
```bash
pytest tests/unit_tests/transformer/moe/test_paged_stashing.py::TestPagedStashing::test_forward_backward_4_layers
```
**Additional context**
Triaged automatically via `/triage-issue`.
Contributor guide
Assessment
This issue has not been assessed yet.