deepseek-ai / deepseek-ai/DeepEP

socketStartConnect: exceeded timeouts (3)socketStartConnect: exceeded timeouts (3)

Open
#86 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

```cpp
:/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketStartConnect:488: /optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketStartConnect:488: socketStartConnect: exceeded timeouts (3)socketStartConnect: exceeded timeouts (3)
:
:/op tnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_send:421: non-zero status: -6
:/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_send:421: non-zero status: -6
:/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:540: non-zero status: -6 /optnvshmem_src_3.1.7-1/src/modules/transport/ibrc/ibrc.cpp:1493: non-zero status: -6 allgather of ep handles failed
:
:/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:540: non-zero status: -6 /optnvshmem_src_3.1.7-1/src/modules/transport/ibrc/ibrc.cpp:1493: non-zero status: -6 allgather of ep handles failed
:
:/optnvshmem_src_3.1.7-1/src/host/transport/transport.cpp:400: non-zero status: 7 /optnvshmem_src_3.1.7-1/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed
:connect EPS failed
:
:
:/optnvshmem_src_3.1.7-1/src/host/init/init.cu:1002: non-zero status: 7 /optnvshmem_src_3.1.7-1/src/host/init/init.cu:1002: non-zero status: 7 nvshmem setup connections failed
:nvshmem setup connections failed
```

你好,请问我跑`python tests/test_low_latency.py`的时候,2机16卡可以,但是换到4机32卡报出这个错误?请问有什么办法吗?
多谢了。

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.