deepseek-ai / deepseek-ai/DeepEP

Can deep_ep run on environments with more than 160 ranks?

Open
#464 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

I noticed a restriction in /csrc/deep_ep.cpp:
EP_HOST_ASSERT(0 <= rank && rank < num_ranks &&
(num_ranks <= NUM_MAX_NVL_PEERS * NUM_MAX_RDMA_PEERS || low_latency_mode));

where NUM_MAX_NVL_PEERS = 8 and NUM_MAX_RDMA_PEERS = 20. This implies the rank count cannot exceed 160 (8*20).

I tested this on a 24-node cluster, and the assertion was triggered. Therefore, my question is:
Does deep_ep actually support training on clusters with more than 20 nodes? Or is there any misunderstanding in my interpretation?

Contributor guide

No contributing guide indexed for this repository

Research direction

Read csrc/deep_ep.cpp, focusing on EP_HOST_ASSERT and the NUM_MAX_NVL_PEERS and NUM_MAX_RDMA_PEERS limits. Reproduce the 24-node configuration or trace how rank counts are validated; done means establishing whether ranks above 160 are supported and documenting the supported behavior or agreed direction.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.