deepseek-ai / deepseek-ai/DeepEP

EPv2: get_rdma_gbs() get half bandwidth in LACP bonding environments

Open
#614 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

Hi,

We found the `get_rdma_gbs()` function in #605 deep_ep/utils/envs.py:246 uses `ibstat` to detect the RDMA NIC bandwidth, which is then used to calculate the number of SMs.

However, in LACP bonding environments, `ibstat` reports only half of the actual bandwidth, leading to incorrect SM calculation.

To avoid this problem, we can set the environment variable (such as `EP_RDMA_GBS` ) to manually specify the correct bandwidth, bypassing `ibstat` detection.

```
diff --git a/deep_ep/utils/envs.py b/deep_ep/utils/envs.py
index f6e34d9..1710c36 100644
--- a/deep_ep/utils/envs.py
+++ b/deep_ep/utils/envs.py
@@ -255,6 +255,10 @@ def get_rdma_gbs(nic_name: str = _DEFAULT_NIC_NAME) -> float:
"""
# noinspection PyBroadException
try:
+ rate = int(os.getenv('EP_RDMA_GBS', 0))
+ if rate != 0:
+ return rate / 8
+
result = subprocess.run(['ibstat'], capture_output=True, text=True, check=True)
output = result.stdout
```

[1.patch](https://github.com/user-attachments/files/27162415/1.patch)

Thank you.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.