NVIDIA / NVIDIA/Megatron-LM

🐛 CI failure: NCCL EP tests abort with pybind11 GIL dec_ref failure

Open
#5,736 0 comments 0 reactions 1 assignee Claimed by @YangFei1990 View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Describe the bug**

NCCL EP unit tests abort in CI with a pybind11 GIL/refcount failure after NCCL EP bootstrap starts. Tag @NVIDIA/mcore-oncall to get oncall attention to this issue.

Failing tests observed in run `29059466380`:

- `tests/unit_tests/transformer/moe/test_token_dispatcher.py::TestFlexDispatcher::test_forward_backward[False-ncclep-False-1-8]`
- `tests/unit_tests/a2a_overlap/test_delay_wgrad_compute.py::TestDelayWgradCompute::test_overlap_dispatch_backward_with_experts_wgrad[fp8_flag0-flex-ncclep-None]`
- `tests/unit_tests/transformer/moe/test_paged_stashing.py::TestNcclEpPagedStashing::test_forward_backward_4_layers`

`Nemo_CICD_Test` failed only as the aggregate gate after the unit-test jobs failed.

**Failing run**

| Field | Value |
|-------|-------|
| PR | [#5680: build: Update Transformer Engine to 2.17](https://github.com/NVIDIA/Megatron-LM/pull/5680) |
| Run | [29059466380](https://github.com/NVIDIA/Megatron-LM/actions/runs/29059466380) |
| Job | [tests/unit_tests/transformer/moe/**/*.py - latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/29059466380/job/86263669371) |
| Job | [tests/unit_tests/a2a_overlap/**/*.py - latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/29059466380/job/86263669405) |
| Job | [tests/unit_tests/**/*.py - gb200 latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/29059466380/job/86263671680) |
| Related TE issue | [NVIDIA/TransformerEngine#3199 comment](https://github.com/NVIDIA/TransformerEngine/issues/3199#issuecomment-4931185365) |

**Error**

```text
NCCL EP: using timeout=101010ms (env=unset, config.timeout_ns=0, source=compile-time default)
pybind11::handle::dec_ref() is being called while the GIL is either not held or invalid.
If you are convinced there is no bug in your code, you can #define PYBIND11_NO_ASSERT_GIL_HELD_INCREF_DECREF to disable this check. In that case you have to ensure this #define is consistently used for all translation units linked into a given pybind11 extension, otherwise there will be ODR violations. The failing pybind11::handle::dec_ref() call was triggered on a torch.dtype object.
terminate called after throwing an instance of std::runtime_error
what(): pybind11::handle::dec_ref() PyGILState_Check() failure.
Fatal Python error: Aborted
...
torch.distributed.elastic.multiprocessing.errors.ChildFailedError
exitcode : -6 (SIGABRT)
```

**Steps/Code to reproduce bug**

Re-run the failing CI jobs linked above, or locally inside the dev container:

```bash
pytest tests/unit_tests/transformer/moe/test_token_dispatcher.py::TestFlexDispatcher::test_forward_backward
pytest tests/unit_tests/a2a_overlap/test_delay_wgrad_compute.py::TestDelayWgradCompute::test_overlap_dispatch_backward_with_experts_wgrad
pytest tests/unit_tests/transformer/moe/test_paged_stashing.py::TestNcclEpPagedStashing::test_forward_backward_4_layers
```

**Additional context**

This is linked to the TE 2.17 issue comment above. The Megatron failures go through NCCL EP bootstrap rather than the Userbuffers `CommOverlapP2P` path from the original TransformerEngine report:

```text
transformer_engine/pytorch/ep.py:125 in ep_bootstrap
megatron/core/transformer/moe/fused_a2a.py:646 in ensure_nccl_ep_bootstrapped
megatron/core/transformer/moe/token_dispatcher.py:1535 in _ensure_bootstrap
```

The affected NCCL EP tests have been quarantined with `pytest.mark.flaky_in_dev` in PR #5680 so the dev CI lane can proceed while this is tracked.

Triaged automatically via `/triage-issue`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.