deepseek-ai / deepseek-ai/DeepEP

[BUG]DeepEP internode timeout

Open
#294 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

4node H100

Image

As shown in image, it stuck here then got "check timeout failed" Error.
*I notice in bottom-left panel, there is a nvshmem log : in nvshmem_finalize*

**num_token 4096 is ok**, but it timeout when num_token=8192/16384

```
set -ex
export NVSHMEM_DEBUG=INFO
export NVSHMEM_IB_ENABLE_IBGDA=true
export NCCL_DEBUG=TRACE
export MASTER_ADDR=10.137.1.127
export MASTER_PORT=5003
export WORLD_SIZE=4
export RANK=0
python test_internode.py
```

code commit: 9ec061204e6763c12a9dd9f4cc5ca3b6c868b552
I can't use later commit because June12's commit 21efbe9b4857ea4d8b0aa0fe8fc5eb673a6a1942 will stuck

ibv_devinfo
```
hca_id: mlx5_ib0
transport: InfiniBand (0)
fw_ver: 28.43.1014
node_guid: 58a2:e103:00e4:2166
sys_image_guid: 58a2:e103:00e4:2166
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000800
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 1
port_lid: 727
port_lmc: 0x00
link_layer: InfiniBand
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.