deepseek-ai / deepseek-ai/DeepEP

NCCL timeout while different ranks execute DeepEP and NCCL communications in different order

Open
#496 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

Reproduced code
```python
if rank % 2 == 0:
with torch.cuda.stream(test_alt_stream):
dist.all_gather_into_tensor(all_topk_idx, topk_idx, group=group)

buffer.low_latency_dispatch(current_x, topk_idx, num_tokens, num_experts,
use_fp8=dispatch_use_fp8, round_scale=round_scale, use_ue8m0=use_ue8m0,
cumulative_local_expert_recv_stats=cumulative_local_expert_recv_stats,
async_finish=not return_recv_hook, return_recv_hook=return_recv_hook)

if rank % 2 != 0:
with torch.cuda.stream(test_alt_stream):
dist.all_gather_into_tensor(all_topk_idx, topk_idx, group=group)
default_stream.wait_stream(test_alt_stream)
```
@sphish mentioned in https://github.com/deepseek-ai/DeepEP/issues/414: ' If each rank executes DeepEP and NCCL communications in the same order, there should be no conflicts. ' What is the reason for this restriction?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.