🐛 CI failure: tests/unit_tests/ssm/test_gated_delta_net.py::TestGatedDeltaNet::test_selective_recompute_gdn (flaky NCCL hang)
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
**Describe the bug**
CI test `tests/unit_tests/ssm/test_gated_delta_net.py` failed in job [`tests/unit_tests/ssm/test_gated_delta_net.py - latest`](https://github.com/NVIDIA/Megatron-LM/actions/runs/28032794934/job/82991762768).
Tag @NVIDIA/mcore-oncall to get oncall's attention to this issue.
The failure is **not** caused by the triggering PR (which only touches DeepSeek-v4 / attention-variant code, unrelated to SSM/gated-delta-net). It is a flaky **NCCL process-group-creation hang** during test fixture setup. The first 8 parametrizations passed; `test_selective_recompute_gdn[2-True-1]` then hung in `initialize_model_parallel` and `rank 4` was aborted with `SIGABRT` (exit code -6) after ~17 min, tearing down the remaining ranks via `SIGTERM`.
**Failing run**
| Field | Value |
|-------|-------|
| PR | [#5011: [dev] [DeepSeek-v4] Packed Sequence (THD) support for DSv4 Hybrid Attention](https://github.com/NVIDIA/Megatron-LM/pull/5011) |
| Run | [28032794934](https://github.com/NVIDIA/Megatron-LM/actions/runs/28032794934) |
| Job | [tests/unit_tests/ssm/test_gated_delta_net.py - latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/28032794934/job/82991762768) |
**Error**
Hang during NCCL communicator creation in test setup (faulthandler stack, most-recent-call-first):
```
File "torch/distributed/distributed_c10d.py", line 2099 in _new_process_group_helper
File "torch/distributed/distributed_c10d.py", line 5565 in _new_group_with_tag
File "torch/distributed/distributed_c10d.py", line 5474 in new_group
File "megatron/core/parallel_state.py", line 240 in create_group
File "megatron/core/parallel_state.py", line 852 in initialize_model_parallel
File "tests/unit_tests/test_utilities.py", line 129 in initialize_model_parallel
File "tests/unit_tests/ssm/test_gated_delta_net.py", line 82 in setup_method
E0623 15:41:34 torch/distributed/elastic/multiprocessing/api.py:988] failed (exitcode: -6) local_rank: 4 (pid: 279)
torch.distributed.elastic.multiprocessing.errors.ChildFailedError
exitcode : -6 (pid: 279) (SIGABRT)
```
Full log available via the job URL above.
**Steps/Code to reproduce bug**
Re-run the failing CI job linked above, or locally inside the dev container:
```bash
pytest tests/unit_tests/ssm/test_gated_delta_net.py
```
**Additional context**
Suspected flaky / infrastructure issue (NCCL init hang on a GPU node), not a code regression. The test is being marked `flaky_in_dev` to unblock CI while the root cause is investigated. Triaged automatically via the `create-issue` skill.
Contributor guide
Assessment
This issue has not been assessed yet.