deepseek-ai / deepseek-ai/DeepEP

test_internode.py runs failed due to "nvshmem detect topo failed"

Open
#355 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

I'm using 2 nodes with 16 H200 GPU to run this test in docker container. And I can run "test_intranode.py" successfuly ,but failed run "test_internode.py".
these are my card status

Image
ibv_devinfo can show ibv devices

```bash
[root:libnvshmem-linux-x86_64-3.3.9_cuda12-archive]$ ibv_devinfo
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 28.42.1000
node_guid: e09d:7303:000d:1fc0
sys_image_guid: e09d:7303:000d:1fc0
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000838
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet

hca_id: mlx5_1
transport: InfiniBand (0)
fw_ver: 28.42.1000
node_guid: e09d:7303:0009:9738
sys_image_guid: e09d:7303:0009:9738
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000838
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet

hca_id: mlx5_2
transport: InfiniBand (0)
fw_ver: 28.42.1000
node_guid: b8e9:2403:002a:5fee
sys_image_guid: b8e9:2403:002a:5fee
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000838
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet

hca_id: mlx5_10
transport: InfiniBand (0)
fw_ver: 28.39.2048
node_guid: 58a2:e103:00b3:2274
sys_image_guid: 58a2:e103:00b3:2274
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000001045
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet

hca_id: mlx5_11
transport: InfiniBand (0)
fw_ver: 28.39.2048
node_guid: 58a2:e103:00b3:2275
sys_image_guid: 58a2:e103:00b3:2274
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000001045
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet

hca_id: mlx5_3
transport: InfiniBand (0)
fw_ver: 28.42.1000
node_guid: e09d:7303:0009:89e0
sys_image_guid: e09d:7303:0009:89e0
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000838
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet

hca_id: mlx5_4
transport: InfiniBand (0)
fw_ver: 28.42.1000
node_guid: b8e9:2403:002a:5f86
sys_image_guid: b8e9:2403:002a:5f86
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000838
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet

hca_id: mlx5_5
transport: InfiniBand (0)
fw_ver: 28.42.1000
node_guid: b8e9:2403:002a:538e
sys_image_guid: b8e9:2403:002a:538e
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000838
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet

hca_id: mlx5_6
transport: InfiniBand (0)
fw_ver: 28.42.1000
node_guid: e09d:7303:0009:6370
sys_image_guid: e09d:7303:0009:6370
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000838
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet

hca_id: mlx5_7
transport: InfiniBand (0)
fw_ver: 28.42.1000
node_guid: b8e9:2403:002d:f8a6
sys_image_guid: b8e9:2403:002d:f8a6
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000838
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet

hca_id: mlx5_bond_0
transport: InfiniBand (0)
fw_ver: 26.43.2026
node_guid: 7c8c:0903:0004:2b52
sys_image_guid: 7c8c:0903:0004:2b52
vendor_id: 0x02c9
vendor_part_id: 4127
hw_ver: 0x0
board_id: MT_0000000531
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
```
here are the script to run test_internode.py
```bash
export NCCL_DEBUG=INFO
export NVSHMEM_DEBUG=TRACE
export NVSHMEM_DEBUG_SUBSYS=ALL
export NVSHMEM_DEBUG_FILE=nvdebug_$RANK

export MASTER_ADDR=${MASTER_ADDR:-localhost}
export RANK=${PET_NODE_RANK:-0}

python3 tests/test_internode.py 2>&1 | tee test_result_internodes_"$RANK".log
```
[test_result_internodes_0.log](https://github.com/user-attachments/files/21678653/test_result_internodes_0.log)

[test_result_internodes_1.log](https://github.com/user-attachments/files/21678719/test_result_internodes_1.log)

running shmem_put_bw on two node with the following scripts:
```shell
export NVSHMEM_DEBUG=TRACE
export NVSHMEM_DEBUG_SUBSYS=ALL
export NVSHMEM_DEBUG_FILE=nvdebug
nvshmrun -n 16 -f libnvshmem-linux-x86_64-3.3.20_cuda12-archive/bin/perftest/device/pt-to-pt/shmem_put_bw 2>&1 | tee put_bw_twonodes.log
```
get these log:

[put_bw_twonodes.log](https://github.com/user-attachments/files/21696166/put_bw_twonodes.log)

[nvdebug.txt](https://github.com/user-attachments/files/21696172/nvdebug.txt)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.