deepseek-ai / deepseek-ai/DeepEP

4 node prefill hang

Open
#326 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

normal combine para:
num_sms = 24
nvl_chunk_size = 1
nvl_buffer_size = 512
rdma_chunk_size = 8
rdma_buffer_size = 128

For my experiment setup:
Four nodes, each with 8 * H20 GPU and 8 * NIC, IB

the logs:

DeepEP combine RDMA receiver timeout, channel: 5, RDMA: 2, nvl: 7, src RDMA: 1, tail: 576, waiting: 4057, expect: 580
DeepEP combine RDMA receiver timeout, channel: 5, RDMA: 2, nvl: 7, src RDMA: 1, tail: 576, waiting: 4054, expect: 577
DeepEP combine RDMA receiver timeout, channel: 5, RDMA: 2, nvl: 7, src RDMA: 1, tail: 576, waiting: 4059, expect: 582
DeepEP combine RDMA receiver timeout, channel: 5, RDMA: 2, nvl: 7, src RDMA: 1, tail: 576, waiting: 4053, expect: 576
DeepEP combine RDMA receiver timeout, channel: 5, RDMA: 2, nvl: 7, src RDMA: 1, tail: 576, waiting: 4055, expect: 578
DeepEP combine RDMA receiver timeout, channel: 5, RDMA: 2, nvl: 7, src RDMA: 1, tail: 576, waiting: 4058, expect: 581
DeepEP combine RDMA receiver timeout, channel: 5, RDMA: 2, nvl: 7, src RDMA: 1, tail: 576, waiting: 4056, expect: 579
DeepEP combine forwarder (RDMA check) timeout, channel: 5, RDMA: 2, nvl: 0, dst RDMA: 1, head: 384, tail: 512, chunked: 8
DeepEP combine forwarder (RDMA check) timeout, channel: 8, RDMA: 2, nvl: 0, dst RDMA: 1, head: 416, tail: 544, chunked: 8
DeepEP combine forwarder (RDMA check) timeout, channel: 6, RDMA: 2, nvl: 0, dst RDMA: 1, head: 416, tail: 544, chunked: 8
DeepEP combine RDMA receiver timeout, channel: 5, RDMA: 2, nvl: 5, src RDMA: 1, tail: 624, waiting: 4097, expect: 633
DeepEP combine RDMA receiver timeout, channel: 5, RDMA: 2, nvl: 5, src RDMA: 1, tail: 624, waiting: 4094, expect: 630

Could you Analyse this issue. Thanks a lot!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.