deepseek-ai / deepseek-ai/DeepEP

AssertionError: No NVLink connection between GPU 0 and GPU 0

Open
#582 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

While running some MoE training examples with Megatron-LM, we have seen below errors:

```
[rank1]: AssertionError: No NVLink connection between GPU 0 and GPU 0, but allow_nvlink_for_normal_mode=True
[rank2]: AssertionError: No NVLink connection between GPU 0 and GPU 0, but allow_nvlink_for_normal_mode=True
```

The topology / `nvidia-smi` output on the compute node:

```
nodes:
GPU0 GPU1 GPU2 GPU3
GPU0 X NV4 NV4 NV4
GPU1 NV4 X NV4 NV4
GPU2 NV4 NV4 X NV4
GPU3 NV4 NV4 NV4 X
```

Looking at the [implementation here](https://github.com/deepseek-ai/DeepEP/blob/567632dd59810d77b3cc05553df953cc0f779799/deep_ep/utils.py#L87), I wonder if we end up in a situation where device ID 0 is duplicated across ranks and the NVLink check end up comparing the same GPU against itself (i.e. [this check](https://github.com/deepseek-ai/DeepEP/blame/567632dd59810d77b3cc05553df953cc0f779799/deep_ep/utils.py#L94) is not sufficient). By the way, here is [the older implementation](https://github.com/deepseek-ai/DeepEP/commit/9ec061204e6763c12a9dd9f4cc5ca3b6c868b552).

cc: @LyricZhao

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.