deepseek-ai / deepseek-ai/DeepEP

When testing with test_internode.py, the dispatch and combine bandwidths are far below expectations.

Open
#336 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

conclusion: After analysis, I believe DeepEP’s high‑throughput kernel is incompatible with the NVSHMEM environment variable NVSHMEM_ENABLE_NIC_PE_MAPPING, causing NVSHMEM to select the same NIC during initialization and thus resulting in overall low test bandwidth.
Environment information: 2 H800 servers, each equipped with 4 IB network cards(CX7).
test command:
node1:
export NVSHMEM_IB_ENABLE_IBGDA=1
export MASTER_ADDR=192.168.0.157
export WORLD_SIZE=2
export RANK=0
export NCCL_SOCKET_IFNAME=bond0
export NVSHMEM_ENABLE_NIC_PE_MAPPING=1
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_6,mlx5_7
export NVSHMEM_HCA_PE_MAPPING="mlx5_0:1:2,mlx5_1:1:2,mlx5_6:1:2,mlx5_7:1:2"
python3 test_internode.py
node2:
export NVSHMEM_IB_ENABLE_IBGDA=1
export MASTER_ADDR=192.168.0.157
export WORLD_SIZE=2
export RANK=1
export NCCL_SOCKET_IFNAME=bond0
export NVSHMEM_ENABLE_NIC_PE_MAPPING=1
export NCCL_IB_HCA=mlx5_0,mlx5_5,mlx5_6,mlx5_9
export NVSHMEM_HCA_PE_MAPPING="mlx5_0:1:2,mlx5_1:1:2,mlx5_6:1:2,mlx5_7:1:2"
python3 test_internode.py
symptoms:
Image
log(high Throughput):
node1:
After inspecting the logs, I found that during NVSHMEM initialization all eight ranks only selected the first NIC to establish connections.
Image
IB NIC traffic monitoring:
node1:
mlx5_0(ibp25s0):
Image
mlx5_1(ibp57s0):

Image
mlx5_6(ibs5):

Image

mlx5_7(ibs7):

Image
Only the first of the four IB NICs is showing any traffic.
code:
DEEPEP:
csrc\deep_ep.cpp
Image
NVSHMEM:

Image
Because DeepEP passes rdma_rank to the NVSHMEM initialization function, NVSHMEM enters the ENABLE_NIC_PE_MAPPING branch during nvshmemi_setup_connections with all eight PEs on the same machine using the same mype_node, causing all traffic to go over a single IB NIC.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.