deepseek-ai / deepseek-ai/DeepEP

H20-3e internode-test error IPC failure

Open
#316 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

IPC failure: Sending data over socket failed: No such file or directory
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/mem/mem_heap.cpp:exchange_heap_memory_handle:775: Fatal IPC Failure

使用sglang 0.4.9 镜像,pip 安装nvshmem,使用了目前最新的master分支 DeepEP

env:
nvidia-cublas-cu12 12.6.4.1
nvidia-cuda-cupti-cu12 12.6.80
nvidia-cuda-nvrtc-cu12 12.6.77
nvidia-cuda-runtime-cu12 12.6.77
nvidia-cudnn-cu12 9.5.1.17
nvidia-cufft-cu12 11.3.0.4
nvidia-cufile-cu12 1.11.1.6
nvidia-curand-cu12 10.3.7.77
nvidia-cusolver-cu12 11.7.1.2
nvidia-cusparse-cu12 12.5.4.2
nvidia-cusparselt-cu12 0.6.3
nvidia-ml-py 12.575.51
nvidia-nccl-cu12 2.26.2
nvidia-nvjitlink-cu12 12.6.85
nvidia-nvshmem-cu12 3.3.9
nvidia-nvtx-cu12 12.6.77
pynvml 12.0.0
python-dotenv 1.1.0

启动命令
MASTER_ADDR=192.168.10.20 MASTER_PORT=10086 WORLD_SIZE=2 RANK=0 python3 tests/test_internode.py
MASTER_ADDR=192.168.10.20 MASTER_PORT=10086 WORLD_SIZE=2 RANK=1 python3 tests/test_internode.py

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with tests/test_internode.py and reproduce the failure using the two listed launch commands, the sglang 0.4.9 image, and the reported package environment. Compare the behavior across both ranks and document the conditions that trigger the IPC error; done means an actionable cause or supported resolution is established.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.