NVIDIA / NVIDIA/Megatron-LM

🐛 CI failure: test_fine_grained_activation_offloading_with_cuda_graph[False-offload_modules1-cuda_graph_scope1-1.0-True]

Open
#4,350 1 comment 0 reactions 1 assignee Claimed by @jingqiny-99 View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Describe the bug**

CI test `tests/unit_tests/pipeline_parallel/test_fine_grained_activation_offloading.py::test_fine_grained_activation_offloading_with_cuda_graph[False-offload_modules1-cuda_graph_scope1-1.0-True]` failed in job [CICD Megatron-LM](https://github.com/NVIDIA/Megatron-LM/actions/runs/24532751669/job/71726804042).
Tag the [@mcore-oncall](https://github.com/orgs/NVIDIA/teams/mcore-oncall) to get oncall's attention to this issue.

**Failing run**

| Field | Value |
|-------|-------|
| PR | [#4332: build: bump TransformerEngine to release_v2.14 (dev)](https://github.com/NVIDIA/Megatron-LM/pull/4332) |
| Run | [24532751669](https://github.com/NVIDIA/Megatron-LM/actions/runs/24532751669) |
| Job | [CICD Megatron-LM](https://github.com/NVIDIA/Megatron-LM/actions/runs/24532751669/job/71726804042) |

**Error**

```
_ test_fine_grained_activation_offloading_with_cuda_graph[False-offload_modules1-cuda_graph_scope1-1.0-True] _

is_mla = False, offload_modules = ['expert_fc1', 'moe_act']
cuda_graph_scope = ['attn', 'moe_router', 'moe_preprocess']
activation_offload_fraction = 1.0, delay_offload = True

def should_bulk_offload(self, name):
"""Determine if the current group should be offloaded."""
assert len(self._groups_to_offload) > 0, "No groups to offload"
group = self.find_group_with_name(self._groups_to_offload, name)
> assert group is not None, f"Group {name} not found in {self._groups_to_offload}"
E AssertionError: Group moe_act not found in []

megatron/core/pipeline_parallel/fine_grained_activation_offload.py:999: AssertionError
```

**Steps/Code to reproduce bug**

Re-run the failing CI job linked above, or locally inside the dev container:

```bash
pytest tests/unit_tests/pipeline_parallel/test_fine_grained_activation_offloading.py::test_fine_grained_activation_offloading_with_cuda_graph[False-offload_modules1-cuda_graph_scope1-1.0-True]
```

**Additional context**

Triaged automatically via `/create-issue`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.