deepseek-ai / deepseek-ai/DeepEP
pip-installing vLLM/torch silently downgrades NCCL below the 2.30.4 floor; import dies at undefined symbol and check_nccl_so() cannot catch it
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
### Problem
`import deep_ep` fails with a raw dynamic-linker error after installing vLLM (or anything that pins
`torch==2.11.0`) into an environment that satisfied the documented NCCL floor:
```
ImportError: /opt/DeepEP/deep_ep/_C.cpython-312-x86_64-linux-gnu.so: undefined symbol: ncclGetLsaDevicePointer
```
The chain (each step verified on H200/EFA, deep_ep 2.0.0, torch 2.11.0+cu130, vllm 0.26.0):
1. The environment starts at `nvidia-nccl-cu13==2.30.4` (the README floor: "NCCL 2.30.4 and above").
2. `pip install vllm==0.26.0` pins `torch==2.11.0`, and **torch 2.11.0 pins `nvidia-nccl-cu13==2.28.9`**
(exact `==`) — pip silently downgrades NCCL as a side effect.
3. NCCL 2.28.x lacks the GIN/LSA device-API symbols `_C.so` was built against, so the next
`import deep_ep` dies at the linker with no version hint.
4. `pip install --no-deps --force-reinstall nvidia-nccl-cu13==2.30.4` fully heals it — deep_ep + vllm
0.26.0 + torch 2.11.0 then import and run together cleanly.
### Why `check_nccl_so()` cannot catch this case
The guard in `deep_ep/__init__.py` compares the **loaded** `libnccl.so` (from `/proc/self/maps`) against
the lib at `find_nccl_root()` — i.e. the NCCL **currently installed**. A wheel downgrade moves *both sides
together*: torch loads 2.28.9 and `find_nccl_root()` now also resolves to 2.28.9. Loaded == reference, the
guard passes, and the failure surfaces one line later at `import deep_ep._C` as an opaque linker error.
The invariant that actually broke is "loaded == what `_C.so` was **built** against", which the build does
not record anywhere the guard can read.
### Suggested fixes (either or both)
1. **Declare the floor in wheel metadata** — `install_requires=["nvidia-nccl-cu13>=2.30.4"]` (per CUDA
variant). pip would then refuse the torch-driven downgrade loudly instead of performing it silently.
Today `setup.py` locates NCCL at build time via `find_pkgs.find_nccl_root()` but ships no constraint.
2. **Record the built-against NCCL version into the wheel** (e.g. write it into the generated `envs.py`
next to the existing persistent envs) and extend `check_nccl_so()` to compare the loaded lib against
*that*, so this failure mode produces the guard's existing clear message instead of
`undefined symbol: ...`.
### Environment
- deep_ep 2.0.0 (wheel built against nvidia-nccl-cu13 2.30.4)
- torch 2.11.0+cu130, vllm 0.26.0, python 3.12
- 2× p5en.48xlarge (H200), EFA — though the failure is host-side and hardware-independent
Happy to provide the full install transcript.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.