deepseek-ai / deepseek-ai/DeepEP
AssertionError: No NVLink connection between GPU 0 and GPU 0
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
While running some MoE training examples with Megatron-LM, we have seen below errors:
```
[rank1]: AssertionError: No NVLink connection between GPU 0 and GPU 0, but allow_nvlink_for_normal_mode=True
[rank2]: AssertionError: No NVLink connection between GPU 0 and GPU 0, but allow_nvlink_for_normal_mode=True
```
The topology / `nvidia-smi` output on the compute node:
```
nodes:
GPU0 GPU1 GPU2 GPU3
GPU0 X NV4 NV4 NV4
GPU1 NV4 X NV4 NV4
GPU2 NV4 NV4 X NV4
GPU3 NV4 NV4 NV4 X
```
Looking at the [implementation here](https://github.com/deepseek-ai/DeepEP/blob/567632dd59810d77b3cc05553df953cc0f779799/deep_ep/utils.py#L87), I wonder if we end up in a situation where device ID 0 is duplicated across ranks and the NVLink check end up comparing the same GPU against itself (i.e. [this check](https://github.com/deepseek-ai/DeepEP/blame/567632dd59810d77b3cc05553df953cc0f779799/deep_ep/utils.py#L94) is not sufficient). By the way, here is [the older implementation](https://github.com/deepseek-ai/DeepEP/commit/9ec061204e6763c12a9dd9f4cc5ca3b6c868b552).
cc: @LyricZhao
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.