deepseek-ai / deepseek-ai/DeepEP

[Question] Unexpectedly low bandwidth when scaling from 2 nodes to 4 nodes

Open
#493 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
2d 17h
Merged PRs (30d)
3

Description

Hi,

I’m testing DeepEP across multiple nodes and found that bandwidth looks normal on 2 nodes, but drops sharply when scaling to 4 nodes.

Could you please suggest possible causes or configurations that might affect this? Any tips for debugging or tuning would be appreciated.

Setup:

GPUs: NVIDIA H200

Network: CX7 400 Gb/s InfiniBand

DeepEP version: a84a248(commit ID)

Test result(only list best):

```
# 2 nodes
[tuning] Best combine: SMs 24, NVL chunk 2, RDMA chunk 20: 44.37 GB/s (RDMA), 145.16 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 28, RDMA chunk 16: 40.89 GB/s (RDMA), 133.78 GB/s (NVL)
[tuning] Best dispatch (FP8): SMs 24, NVL chunk 28, RDMA chunk 24: 38.19 GB/s (RDMA), 124.92 GB/s (NVL)

# 4 nodes
[tuning] Best combine: SMs 24, NVL chunk 2, RDMA chunk 8: 12.53 GB/s (RDMA), 25.05 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 12, RDMA chunk 4: 12.27 GB/s (RDMA), 24.51 GB/s (NVL)
[tuning] Best dispatch (FP8): SMs 24, NVL chunk 4, RDMA chunk 4: 13.15 GB/s (RDMA), 26.27 GB/s (NVL)
```

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by reproducing the reported DeepEP tuning runs with the listed H200 and CX7 InfiniBand setup, comparing the two-node and four-node results. Done means identifying the cause of the scaling drop and documenting a verified configuration or debugging direction.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems, networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.