deepseek-ai / deepseek-ai/DeepEP
[Question] Unexpectedly low bandwidth when scaling from 2 nodes to 4 nodes
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 3
Description
Hi,
I’m testing DeepEP across multiple nodes and found that bandwidth looks normal on 2 nodes, but drops sharply when scaling to 4 nodes.
Could you please suggest possible causes or configurations that might affect this? Any tips for debugging or tuning would be appreciated.
Setup:
GPUs: NVIDIA H200
Network: CX7 400 Gb/s InfiniBand
DeepEP version: a84a248(commit ID)
Test result(only list best):
```
# 2 nodes
[tuning] Best combine: SMs 24, NVL chunk 2, RDMA chunk 20: 44.37 GB/s (RDMA), 145.16 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 28, RDMA chunk 16: 40.89 GB/s (RDMA), 133.78 GB/s (NVL)
[tuning] Best dispatch (FP8): SMs 24, NVL chunk 28, RDMA chunk 24: 38.19 GB/s (RDMA), 124.92 GB/s (NVL)
# 4 nodes
[tuning] Best combine: SMs 24, NVL chunk 2, RDMA chunk 8: 12.53 GB/s (RDMA), 25.05 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 12, RDMA chunk 4: 12.27 GB/s (RDMA), 24.51 GB/s (NVL)
[tuning] Best dispatch (FP8): SMs 24, NVL chunk 4, RDMA chunk 4: 13.15 GB/s (RDMA), 26.27 GB/s (NVL)
```
Thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by reproducing the reported DeepEP tuning runs with the listed H200 and CX7 InfiniBand setup, comparing the two-node and four-node results. Done means identifying the cause of the scaling drop and documenting a verified configuration or debugging direction.
Written by the indexing model from the issue text.
Assessment
- Domain
- distributed-systems, networking, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100