deepseek-ai / deepseek-ai/DeepEP
EPv2: get_rdma_gbs() get half bandwidth in LACP bonding environments
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
Hi,
We found the `get_rdma_gbs()` function in #605 deep_ep/utils/envs.py:246 uses `ibstat` to detect the RDMA NIC bandwidth, which is then used to calculate the number of SMs.
However, in LACP bonding environments, `ibstat` reports only half of the actual bandwidth, leading to incorrect SM calculation.
To avoid this problem, we can set the environment variable (such as `EP_RDMA_GBS` ) to manually specify the correct bandwidth, bypassing `ibstat` detection.
```
diff --git a/deep_ep/utils/envs.py b/deep_ep/utils/envs.py
index f6e34d9..1710c36 100644
--- a/deep_ep/utils/envs.py
+++ b/deep_ep/utils/envs.py
@@ -255,6 +255,10 @@ def get_rdma_gbs(nic_name: str = _DEFAULT_NIC_NAME) -> float:
"""
# noinspection PyBroadException
try:
+ rate = int(os.getenv('EP_RDMA_GBS', 0))
+ if rate != 0:
+ return rate / 8
+
result = subprocess.run(['ibstat'], capture_output=True, text=True, check=True)
output = result.stdout
```
[1.patch](https://github.com/user-attachments/files/27162415/1.patch)
Thank you.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.