deepseek-ai / deepseek-ai/DeepEP
H20-3e internode-test error IPC failure
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
IPC failure: Sending data over socket failed: No such file or directory
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/mem/mem_heap.cpp:exchange_heap_memory_handle:775: Fatal IPC Failure
使用sglang 0.4.9 镜像,pip 安装nvshmem,使用了目前最新的master分支 DeepEP
env:
nvidia-cublas-cu12 12.6.4.1
nvidia-cuda-cupti-cu12 12.6.80
nvidia-cuda-nvrtc-cu12 12.6.77
nvidia-cuda-runtime-cu12 12.6.77
nvidia-cudnn-cu12 9.5.1.17
nvidia-cufft-cu12 11.3.0.4
nvidia-cufile-cu12 1.11.1.6
nvidia-curand-cu12 10.3.7.77
nvidia-cusolver-cu12 11.7.1.2
nvidia-cusparse-cu12 12.5.4.2
nvidia-cusparselt-cu12 0.6.3
nvidia-ml-py 12.575.51
nvidia-nccl-cu12 2.26.2
nvidia-nvjitlink-cu12 12.6.85
nvidia-nvshmem-cu12 3.3.9
nvidia-nvtx-cu12 12.6.77
pynvml 12.0.0
python-dotenv 1.1.0
启动命令
MASTER_ADDR=192.168.10.20 MASTER_PORT=10086 WORLD_SIZE=2 RANK=0 python3 tests/test_internode.py
MASTER_ADDR=192.168.10.20 MASTER_PORT=10086 WORLD_SIZE=2 RANK=1 python3 tests/test_internode.py
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with tests/test_internode.py and reproduce the failure using the two listed launch commands, the sglang 0.4.9 image, and the reported package environment. Compare the behavior across both ranks and document the conditions that trigger the IPC error; done means an actionable cause or supported resolution is established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100