deepseek-ai / deepseek-ai/DeepEP

H20's internode RDMA & NVL bandwidth drops to 50% after pr 245

Open
#361 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

### Environment

- Hardware: 2 nodes × 8× H20 GPUs,
- Network: RoCE, CX6 InfiniBand NICs

### Problem

Internode RDMA bandwidth dropped by ~50% after commit `bc118b` ( pr #245 ) , confirmed via bisect.

the commit before `bc118b` (`9eb2f`)
```shell
[tuning] Best dispatch (FP8): SMs 24, NVL chunk 24, RDMA chunk 4: 44.00 GB/s (RDMA), 144.10 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 4, RDMA chunk 8: 45.35 GB/s (RDMA), 148.50 GB/s (NVL)
[tuning] Best combine: SMs 24, NVL chunk 1, RDMA chunk 8: 43.93 GB/s (RDMA), 143.85 GB/s (NVL)
```
after commit `bc118b`
```shell
[tuning] Best dispatch (FP8): SMs 24, NVL chunk 8, RDMA chunk 12: 23.18 GB/s (RDMA), 75.91 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 4, RDMA chunk 12: 23.72 GB/s (RDMA), 77.68 GB/s (NVL)
[tuning] Best combine: SMs 24, NVL chunk 1, RDMA chunk 8: 23.41 GB/s (RDMA), 76.65 GB/s (NVL)
```

I have reviewed the modifications in the submission, but I find them too challenging to grasp. Could the developers reproduce and resolve this issue?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.