deepseek-ai / deepseek-ai/DeepEP

deep_ep.cpp:200 init fails on partial CUDA peer access; preflight and report unsupported device pairs (cudaErrorPeerAccessUnsupported)

Open
#584 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

## Problem

> **Scope:** Not requesting PCIe-only support; proposing an early capability check and a clearer
> error message when the NVLink/full-mesh P2P requirement isn't met.

DeepEP initialization (`runtime.sync(...)` in `deep_ep.cpp:200`) aborts with a CUDA peer access
error on an 8-GPU PCIe-only host where only 4 undirected GPU pairs support
`cudaDeviceCanAccessPeer`. The error fires on the first unsupported pair without validating the
full required set upfront or reporting which pairs are incompatible. (The SGLang wrapper path that
triggers this is shown in the stack trace under Evidence.)

```text
RuntimeError: Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:200
'peer access is not supported between these two devices'
```

A secondary CUDA error at `runtime.cu:29` also appears in the exception chain — see crash
excerpt for details; its relationship to the primary peer-access failure is unclear.

## Steps to reproduce

1. Use an 8-GPU host with partial CUDA P2P: only same-PCIe-switch pairs support peer access
(e.g., `{0,1}`, `{2,3}`, `{4,5}`, `{6,7}`). All other 48 of 56 off-diagonal pairs return
`cudaErrorPeerAccessUnsupported` (err=217).

2. Launch with EP=8 and DeepEP normal mode:

```bash
python3 -m sglang.launch_server \
--model-path Qwen/Qwen3.5-397B-A17B-FP8 \
--tp-size 8 \
--ep-size 8 \
--moe-a2a-backend deepep \
--deepep-mode normal \
--disable-cuda-graph
```

> Tested with `Qwen/Qwen3.5-397B-A17B-FP8`; any MoE model with EP support should reproduce on a
> partial-P2P topology.

> **Note:** `--deepep-mode normal` is required to isolate this crash. `--deepep-mode auto` fails
> earlier with an unrelated `deep_gemm` assertion.

3. All 8 workers crash on first token dispatch during server warmup.

> The repro command above is minimal. The exact command used in our run (including additional
> flags such as `--kv-cache-dtype`, `--mem-fraction-static`, etc.) is in the Environment section.

## Evidence

On this host, 4 undirected PIX pairs support P2P (`{0,1}`, `{2,3}`, `{4,5}`, `{6,7}`); in the
directed 8×8 matrix this appears as 8/56 off-diagonal 'Y' entries, and the remaining directed
pairs fail with `cudaErrorPeerAccessUnsupported` (217).

nvidia-smi topo -m (8-GPU PCIe topology, no NVLink)

```text
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity
GPU0 X PIX NODE NODE SYS SYS SYS SYS 0-47,96-143 0
GPU1 PIX X NODE NODE SYS SYS SYS SYS 0-47,96-143 0
GPU2 NODE NODE X PIX SYS SYS SYS SYS 0-47,96-143 0
GPU3 NODE NODE PIX X SYS SYS SYS SYS 0-47,96-143 0
GPU4 SYS SYS SYS SYS X PIX NODE NODE 48-95,144-191 1
GPU5 SYS SYS SYS SYS PIX X NODE NODE 48-95,144-191 1
GPU6 SYS SYS SYS SYS NODE NODE X PIX 48-95,144-191 1
GPU7 SYS SYS SYS SYS NODE NODE PIX X 48-95,144-191 1

Legend: PIX = single PCIe bridge NODE = within NUMA SYS = cross-NUMA NV# = NVLink (none present)
```

CUDA peer access matrix — cudaDeviceCanAccessPeer (8×8)

```text
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7
GPU0 X Y N N N N N N
GPU1 Y X N N N N N N
GPU2 N N X Y N N N N
GPU3 N N Y X N N N N
GPU4 N N N N X Y N N
GPU5 N N N N Y X N N
GPU6 N N N N N N X Y
GPU7 N N N N N N Y X

Summary: 4 undirected PIX pairs support P2P ({0,1}, {2,3}, {4,5}, {6,7}); in the directed
8×8 matrix this appears as 8/56 off-diagonal 'Y' entries, and the remaining directed pairs
fail with cudaErrorPeerAccessUnsupported (217).
```

Crash log excerpt

```text
[2026-02-19 21:47:12 TP0 EP0] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.
[2026-02-19 21:47:12 TP1 EP1] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.
[2026-02-19 21:47:12 TP2 EP2] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.
[2026-02-19 21:47:12 TP3 EP3] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.
[2026-02-19 21:47:12 TP4 EP4] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.
[2026-02-19 21:47:12 TP5 EP5] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.
[2026-02-19 21:47:12 TP6 EP6] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.
[2026-02-19 21:47:12 TP7 EP7] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.

File ".../sglang/srt/models/qwen2_moe.py", line 271, in _forward_deepep
File ".../sglang/srt/batch_overlap/two_batch_overlap.py", line 1035, in dispatch
File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 782, in dispatch
File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 801, in dispatch_b
File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 410, in dispatch_b
File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 524, in _get_buffer
return DeepEPBuffer.get_deepep_buffer(
File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 221, in get_deepep_buffer
RuntimeError: Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:200
'peer access is not supported between these two devices'
what(): Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/runtime.cu:29
'named symbol not found'
```

## Request

Two improvements that would help users on non-NVLink / partial-P2P hardware:

**1. Document the peer-access assumption.**
It would help to know whether DeepEP requires full-mesh peer access across all intranode EP devices,
or whether partial P2P is supported in some configurations. If full-mesh is required, noting this
in the README would let users rule out DeepEP before attempting it on PCIe hosts.

**2. Validate all required pairs at init, report them together.**
Rather than calling `runtime.sync` and aborting on the first unsupported pair, a preflight loop
over all required pairs would let users understand the full scope of incompatibility:

```text
DeepEP P2P preflight failed.
Unsupported pairs: (GPU0,GPU2), (GPU0,GPU3), ... (48/56 directed pairs).
DeepEP requires full CUDA peer access across all participating devices.
Try a different MoE A2A backend or a topology with full P2P support.
```

The "Only use 20 SMs" warning already fires before the crash, so partial connectivity is detectable
at this stage. An actionable error at the same point — before any token dispatch — would improve
the experience significantly.

## Environment

```
DeepEP: compiled package bundled with lmsysorg/sglang:nightly-dev-cu13-20260219-46226798
(no separate git SHA; /sgl-workspace/DeepEP is not a git repo in this container)
Python: 3.12.3
torch: 2.9.1+cu130
CUDA toolkit: 13.0 (nvcc V13.0.88)
Driver: 580.126.09
GPU: 8x NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120, CC 12.0, 96 GB)
OS: Ubuntu 22.04 / kernel 6.8.0-1044-aws
Env: SGLANG_DISABLE_DEEP_GEMM=1 (SM120-specific; required on this hardware to use
--deepep-mode normal; not needed for reproduction on hosts where deep_gemm works)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.