deepseek-ai / deepseek-ai/DeepEP
[BUG]DeepEP internode timeout
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
4node H100
As shown in image, it stuck here then got "check timeout failed" Error.
*I notice in bottom-left panel, there is a nvshmem log : in nvshmem_finalize*
**num_token 4096 is ok**, but it timeout when num_token=8192/16384
```
set -ex
export NVSHMEM_DEBUG=INFO
export NVSHMEM_IB_ENABLE_IBGDA=true
export NCCL_DEBUG=TRACE
export MASTER_ADDR=10.137.1.127
export MASTER_PORT=5003
export WORLD_SIZE=4
export RANK=0
python test_internode.py
```
code commit: 9ec061204e6763c12a9dd9f4cc5ca3b6c868b552
I can't use later commit because June12's commit 21efbe9b4857ea4d8b0aa0fe8fc5eb673a6a1942 will stuck
ibv_devinfo
```
hca_id: mlx5_ib0
transport: InfiniBand (0)
fw_ver: 28.43.1014
node_guid: 58a2:e103:00e4:2166
sys_image_guid: 58a2:e103:00e4:2166
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000800
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 1
port_lid: 727
port_lmc: 0x00
link_layer: InfiniBand
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.