deepseek-ai / deepseek-ai/DeepEP

pip-installing vLLM/torch silently downgrades NCCL below the 2.30.4 floor; import dies at undefined symbol and check_nccl_so() cannot catch it

Open
#710 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

### Problem

`import deep_ep` fails with a raw dynamic-linker error after installing vLLM (or anything that pins
`torch==2.11.0`) into an environment that satisfied the documented NCCL floor:

```
ImportError: /opt/DeepEP/deep_ep/_C.cpython-312-x86_64-linux-gnu.so: undefined symbol: ncclGetLsaDevicePointer
```

The chain (each step verified on H200/EFA, deep_ep 2.0.0, torch 2.11.0+cu130, vllm 0.26.0):

1. The environment starts at `nvidia-nccl-cu13==2.30.4` (the README floor: "NCCL 2.30.4 and above").
2. `pip install vllm==0.26.0` pins `torch==2.11.0`, and **torch 2.11.0 pins `nvidia-nccl-cu13==2.28.9`**
(exact `==`) — pip silently downgrades NCCL as a side effect.
3. NCCL 2.28.x lacks the GIN/LSA device-API symbols `_C.so` was built against, so the next
`import deep_ep` dies at the linker with no version hint.
4. `pip install --no-deps --force-reinstall nvidia-nccl-cu13==2.30.4` fully heals it — deep_ep + vllm
0.26.0 + torch 2.11.0 then import and run together cleanly.

### Why `check_nccl_so()` cannot catch this case

The guard in `deep_ep/__init__.py` compares the **loaded** `libnccl.so` (from `/proc/self/maps`) against
the lib at `find_nccl_root()` — i.e. the NCCL **currently installed**. A wheel downgrade moves *both sides
together*: torch loads 2.28.9 and `find_nccl_root()` now also resolves to 2.28.9. Loaded == reference, the
guard passes, and the failure surfaces one line later at `import deep_ep._C` as an opaque linker error.
The invariant that actually broke is "loaded == what `_C.so` was **built** against", which the build does
not record anywhere the guard can read.

### Suggested fixes (either or both)

1. **Declare the floor in wheel metadata** — `install_requires=["nvidia-nccl-cu13>=2.30.4"]` (per CUDA
variant). pip would then refuse the torch-driven downgrade loudly instead of performing it silently.
Today `setup.py` locates NCCL at build time via `find_pkgs.find_nccl_root()` but ships no constraint.
2. **Record the built-against NCCL version into the wheel** (e.g. write it into the generated `envs.py`
next to the existing persistent envs) and extend `check_nccl_so()` to compare the loaded lib against
*that*, so this failure mode produces the guard's existing clear message instead of
`undefined symbol: ...`.

### Environment

- deep_ep 2.0.0 (wheel built against nvidia-nccl-cu13 2.30.4)
- torch 2.11.0+cu130, vllm 0.26.0, python 3.12
- 2× p5en.48xlarge (H200), EFA — though the failure is host-side and hardware-independent

Happy to provide the full install transcript.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.