deepseek-ai / deepseek-ai/DeepEP
socketStartConnect: exceeded timeouts (3)socketStartConnect: exceeded timeouts (3)
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
```cpp
:/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketStartConnect:488: /optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketStartConnect:488: socketStartConnect: exceeded timeouts (3)socketStartConnect: exceeded timeouts (3)
:
:/op tnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_send:421: non-zero status: -6
:/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_send:421: non-zero status: -6
:/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:540: non-zero status: -6 /optnvshmem_src_3.1.7-1/src/modules/transport/ibrc/ibrc.cpp:1493: non-zero status: -6 allgather of ep handles failed
:
:/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:540: non-zero status: -6 /optnvshmem_src_3.1.7-1/src/modules/transport/ibrc/ibrc.cpp:1493: non-zero status: -6 allgather of ep handles failed
:
:/optnvshmem_src_3.1.7-1/src/host/transport/transport.cpp:400: non-zero status: 7 /optnvshmem_src_3.1.7-1/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed
:connect EPS failed
:
:
:/optnvshmem_src_3.1.7-1/src/host/init/init.cu:1002: non-zero status: 7 /optnvshmem_src_3.1.7-1/src/host/init/init.cu:1002: non-zero status: 7 nvshmem setup connections failed
:nvshmem setup connections failed
```
你好,请问我跑`python tests/test_low_latency.py`的时候,2机16卡可以,但是换到4机32卡报出这个错误?请问有什么办法吗?
多谢了。
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.