deepseek-ai / deepseek-ai/DeepEP
Setting deepep to low_latency mode fails to start model serving
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
When I set the deepep mode to auto and normal, the stress test is normal, but when I compare the low_latency mode, the following error occurs. `RuntimeError: Failed: Assertion error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:1025 'x.size(0) == topk_idx.size(0) and x.size(0) <= num_max_dispatch_tokens_per_rank' `Has anyone encountered this before? What should I do to solve it?
```
[2025-05-14 03:58:33 DP0 TP0] TpModelWorkerClient hit an exception: Traceback (most recent call last):
File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 116, in forward_thread_func
self.forward_thread_func_()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 147, in forward_thread_func_
logits_output, next_token_ids = self.worker.forward_batch_generation(
File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 181, in forward_batch_generation
logits_output = self.model_runner.forward(forward_batch)
File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1122, in forward
return self._forward_raw(forward_batch, skip_attn_backend_init)
File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1155, in _forward_raw
return self.forward_extend(
File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1064, in forward_extend
return self.model.forward(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 2099, in forward
hidden_states = self.model(input_ids, positions, forward_batch, input_embeds)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1916, in forward
hidden_states, residual = layer(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1462, in forward
return self.forward_ffn_with_scattered_input(
File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1617, in forward_ffn_with_scattered_input
hidden_states = self.mlp(hidden_states, forward_batch.forward_mode)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 355, in forward
return self.forward_deepep(hidden_states, forward_mode)
File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 386, in forward_deepep
self._forward_deepep_dispatch_a(
File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 466, in _forward_deepep_dispatch_a
chosen_deepep_dispatcher.dispatch_a(
File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/ep_moe/token_dispatcher.py", line 713, in dispatch_a
inner_state = self._get_impl(forward_mode).dispatch_a(
File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/ep_moe/token_dispatcher.py", line 519, in dispatch_a
hidden_states, masked_m, event, hook = self._dispatch_core(
File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/ep_moe/token_dispatcher.py", line 603, in _dispatch_core
buffer.low_latency_dispatch(
File "/usr/local/lib/python3.10/dist-packages/deep_ep/buffer.py", line 487, in low_latency_dispatch
self.runtime.low_latency_dispatch(x, topk_idx,
RuntimeError: Failed: Assertion error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:1025 'x.size(0) == topk_idx.size(0) and x.size(0) <= num_max_dispatch_tokens_per_rank'
[2025-05-14 03:58:33] Child process unexpectedly failed with an exit code 131. pid=255
[2025-05-14 03:58:34] Child process unexpectedly failed with an exit code 9. pid=1401
[2025-05-14 03:58:34] Child process unexpectedly failed with an exit code 9. pid=2194
[2025-05-14 03:58:34] Child process unexpectedly failed with an exit code 9. pid=2198
```
cmd
```
export SGL_ENABLE_JIT_DEEPGEMM=1
python3 -m sglang.launch_server \
--model $model \
--trust-remote-code \
--tp 8 \
--dp 8 \
--enable-dp-attention \
--enable-deepep-moe \
--deepep-mode low_latency \
--enable-metrics
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.