NVIDIA-NeMo / NVIDIA-NeMo/RL

DeepEP 17cf pin times out in multi-node normal dispatch with NVSHMEM 3.7.2

Open
#4,027 0 comments 0 reactions 1 assignee Claimed by @seonjinn View on GitHub
bug Speed
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

## Summary

NeMo RL currently pins DeepEP to `17cfb817bccec3a9c247013360cc550c2bac441e` while allowing `nvidia-nvshmem-cu13>=3.6.5`. In the current nightly environment, this resolves to NVSHMEM 3.7.2.

With this combination, multi-node DeepEP normal dispatch consistently reaches its 100-second CPU timeout. The failure was first observed during the initial MoE policy logprob forward for Gemma 4 26B with EP16/CP2. The exact same error also occurs in two independent Qwen3.5 35B-A3B EP16 nightly tests, indicating a dependency-level multi-node DeepEP problem rather than a Gemma-specific issue.

Relevant dependency declarations on current `main`:

- [DeepEP pin](https://github.com/NVIDIA-NeMo/RL/blob/a366bc8cffec730080354415d74c2c12462ea028/pyproject.toml#L148)
- [NVSHMEM requirement](https://github.com/NVIDIA-NeMo/RL/blob/a366bc8cffec730080354415d74c2c12462ea028/pyproject.toml#L62)

## Observed error

The timeout occurs in the first inter-node dispatch:

```text
GroupedExpertsDeepEP.forward
-> token_permutation2
-> dispatch_all_to_all
-> fused_dispatch
-> deep_ep.Buffer.internode_dispatch

RuntimeError: DeepEP error: timeout (dispatch CPU)
```

Cleanup after the timeout can also report a CUDA illegal-memory-access error, but that appears downstream of the dispatch timeout.

## Nightly CI failures

[NVIDIA nightly pipeline 66217369](https://gitlab-master.nvidia.com/dl/JoC/nemo-ci/-/pipelines/66217369) contains two independent Qwen3.5 35B-A3B EP16 failures with the same signature:

- [LLM GRPO Qwen3.5 35B-A3B, 2 nodes, Automodel EP16 — job 425679893](https://gitlab-master.nvidia.com/dl/JoC/nemo-ci/-/jobs/425679893)
- [VLM GRPO Qwen3.5 35B-A3B Geo3K, 2 nodes, Automodel EP16 — job 425679909](https://gitlab-master.nvidia.com/dl/JoC/nemo-ci/-/jobs/425679909)

Both traces follow the same path through Automodel's `fused_a2a.py`, `buffer.dispatch`, and `Buffer.internode_dispatch` before raising `RuntimeError: DeepEP error: timeout (dispatch CPU)`.

## Controlled reproduction

We ran a minimal EP16 normal-dispatch and combine test using:

- Two H100 nodes, eight GPUs per node
- PyTorch 2.11.0+cu130
- CUDA 13.0
- NVSHMEM 3.7.2
- Gemma 4 26B routing dimensions:
- hidden size: 2816
- experts: 128
- top-k: 8
- tokens: 4096

The same container, environment, and test were used across separate H100 allocations.

| DeepEP implementation | Result |
|---|---|
| Current pin `17cfb81` | Failed after approximately 100 seconds with `timeout (dispatch CPU)` |
| Previous pin `29d31c0` containing DeepEP #564 | Passed; dispatch completed in 6.57 ms |
| Current pin `17cfb81` plus only the two #564 compatibility changes | Passed; dispatch completed in 9.60 ms |

The passing cases also completed the combine and correctness checks without CUDA, NCCL, assertion, or timeout errors.

NVSHMEM selected the `ibrc` transport and initialized IBGDA successfully in all three cases, so the failure does not appear to be an initial transport or GID configuration problem.

## Root cause

The result is experimentally isolated to the NVSHMEM compatibility changes from [DeepEP #564](https://github.com/deepseek-ai/DeepEP/pull/564):

1. Select the appropriate RC QP representation for the newer NVSHMEM internal layout.
2. Enable relocatable device code when compiling with NVSHMEM, preventing duplicate device-state definitions.

The current `17cfb81` hybrid-ep lineage does not contain these changes. Manually porting only those changes onto that revision makes the same test pass.

This is consistent with:

- [DeepEP #553](https://github.com/deepseek-ai/DeepEP/issues/553), where the same dispatch timeout was reported and #564 was confirmed as the fix.
- [DeepEP #707](https://github.com/deepseek-ai/DeepEP/issues/707), reporting legacy normal-mode dispatch timeouts with newer NVSHMEM.
- [DeepEP #696](https://github.com/deepseek-ai/DeepEP/pull/696), which also discusses incompatible NVSHMEM runtime layouts.

The regression entered NeMo RL when [PR #3436](https://github.com/NVIDIA-NeMo/RL/pull/3436) changed the x86 DeepEP pin from `29d31c0` to `17cfb81`.

## Proposed fix

- Port the #564-equivalent RC QP indexing and RDC fixes onto the pinned hybrid-ep revision, or update the pin to a hybrid-ep revision containing equivalent compatibility changes.
- Consider pinning NVSHMEM to an explicitly tested version or range instead of leaving the upper bound open.
- Add a two-node normal DeepEP dispatch smoke test to dependency or nightly validation. Single-node initialization does not expose this failure.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.