deepseek-ai / deepseek-ai/DeepEP
H20's internode RDMA & NVL bandwidth drops to 50% after pr 245
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
### Environment
- Hardware: 2 nodes × 8× H20 GPUs,
- Network: RoCE, CX6 InfiniBand NICs
### Problem
Internode RDMA bandwidth dropped by ~50% after commit `bc118b` ( pr #245 ) , confirmed via bisect.
the commit before `bc118b` (`9eb2f`)
```shell
[tuning] Best dispatch (FP8): SMs 24, NVL chunk 24, RDMA chunk 4: 44.00 GB/s (RDMA), 144.10 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 4, RDMA chunk 8: 45.35 GB/s (RDMA), 148.50 GB/s (NVL)
[tuning] Best combine: SMs 24, NVL chunk 1, RDMA chunk 8: 43.93 GB/s (RDMA), 143.85 GB/s (NVL)
```
after commit `bc118b`
```shell
[tuning] Best dispatch (FP8): SMs 24, NVL chunk 8, RDMA chunk 12: 23.18 GB/s (RDMA), 75.91 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 4, RDMA chunk 12: 23.72 GB/s (RDMA), 77.68 GB/s (NVL)
[tuning] Best combine: SMs 24, NVL chunk 1, RDMA chunk 8: 23.41 GB/s (RDMA), 76.65 GB/s (NVL)
```
I have reviewed the modifications in the submission, but I find them too challenging to grasp. Could the developers reproduce and resolve this issue?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.