NVIDIA / NVIDIA/Megatron-LM

🐛 CI failure: tests/unit_tests/ssm/test_gated_delta_net.py::TestGatedDeltaNet::test_selective_recompute_gdn (flaky NCCL hang)

Open
#5,473 0 comments 0 reactions 1 assignee Claimed by @yuzhongw-nvidia View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Describe the bug**

CI test `tests/unit_tests/ssm/test_gated_delta_net.py` failed in job [`tests/unit_tests/ssm/test_gated_delta_net.py - latest`](https://github.com/NVIDIA/Megatron-LM/actions/runs/28032794934/job/82991762768).
Tag @NVIDIA/mcore-oncall to get oncall's attention to this issue.

The failure is **not** caused by the triggering PR (which only touches DeepSeek-v4 / attention-variant code, unrelated to SSM/gated-delta-net). It is a flaky **NCCL process-group-creation hang** during test fixture setup. The first 8 parametrizations passed; `test_selective_recompute_gdn[2-True-1]` then hung in `initialize_model_parallel` and `rank 4` was aborted with `SIGABRT` (exit code -6) after ~17 min, tearing down the remaining ranks via `SIGTERM`.

**Failing run**

| Field | Value |
|-------|-------|
| PR | [#5011: [dev] [DeepSeek-v4] Packed Sequence (THD) support for DSv4 Hybrid Attention](https://github.com/NVIDIA/Megatron-LM/pull/5011) |
| Run | [28032794934](https://github.com/NVIDIA/Megatron-LM/actions/runs/28032794934) |
| Job | [tests/unit_tests/ssm/test_gated_delta_net.py - latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/28032794934/job/82991762768) |

**Error**

Hang during NCCL communicator creation in test setup (faulthandler stack, most-recent-call-first):

```
File "torch/distributed/distributed_c10d.py", line 2099 in _new_process_group_helper
File "torch/distributed/distributed_c10d.py", line 5565 in _new_group_with_tag
File "torch/distributed/distributed_c10d.py", line 5474 in new_group
File "megatron/core/parallel_state.py", line 240 in create_group
File "megatron/core/parallel_state.py", line 852 in initialize_model_parallel
File "tests/unit_tests/test_utilities.py", line 129 in initialize_model_parallel
File "tests/unit_tests/ssm/test_gated_delta_net.py", line 82 in setup_method

E0623 15:41:34 torch/distributed/elastic/multiprocessing/api.py:988] failed (exitcode: -6) local_rank: 4 (pid: 279)
torch.distributed.elastic.multiprocessing.errors.ChildFailedError
exitcode : -6 (pid: 279) (SIGABRT)
```

Full log available via the job URL above.

**Steps/Code to reproduce bug**

Re-run the failing CI job linked above, or locally inside the dev container:

```bash
pytest tests/unit_tests/ssm/test_gated_delta_net.py
```

**Additional context**

Suspected flaky / infrastructure issue (NCCL init hang on a GPU node), not a code regression. The test is being marked `flaky_in_dev` to unblock CI while the root cause is investigated. Triaged automatically via the `create-issue` skill.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.