deepseek-ai / deepseek-ai/DeepEP
NCCL timeout while different ranks execute DeepEP and NCCL communications in different order
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
Reproduced code
```python
if rank % 2 == 0:
with torch.cuda.stream(test_alt_stream):
dist.all_gather_into_tensor(all_topk_idx, topk_idx, group=group)
buffer.low_latency_dispatch(current_x, topk_idx, num_tokens, num_experts,
use_fp8=dispatch_use_fp8, round_scale=round_scale, use_ue8m0=use_ue8m0,
cumulative_local_expert_recv_stats=cumulative_local_expert_recv_stats,
async_finish=not return_recv_hook, return_recv_hook=return_recv_hook)
if rank % 2 != 0:
with torch.cuda.stream(test_alt_stream):
dist.all_gather_into_tensor(all_topk_idx, topk_idx, group=group)
default_stream.wait_stream(test_alt_stream)
```
@sphish mentioned in https://github.com/deepseek-ai/DeepEP/issues/414: ' If each rank executes DeepEP and NCCL communications in the same order, there should be no conflicts. ' What is the reason for this restriction?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.