kvcache-ai / kvcache-ai/ktransformers
支持 SGLang 的 expert-parallel-size(EP)
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Reminder
- [x] I have read the above rules and searched the existing issues.
### Description
目前 kt_ep_wrapper 不支持 SGLang 的 --expert-parallel-size 参数。
TP=8 在 8×L20(无 NVLink)下 PCIe all-reduce 是主要瓶颈,
改用 TP=4+EP=2 通信开销能明显降低。
当前问题
配了 --tensor-parallel-size 4 --expert-parallel-size 2 后,
模型加载阶段崩溃:
RuntimeError: The size of tensor a (192) must match the size of
tensor b (384) at non-singleton dimension 0
gpu_experts_mask 初始化时按 384 个全量 experts 创建,
但 SGLang EP 切分后每个 rank 只剩 192 个 experts,对不上。
期望行为
kt_ep_wrapper 感知 SGLang 的 EP rank,按 EP 切分后的 local expert 数
来创建 gpu_experts_mask,--kt-num-gpu-experts 也应该按 EP rank 自适应。
环境
- 模型:Kimi-K2.6 RAWINT4
- GPU:8×L20(SM89,无 NVLink)
- CPU:双路 Xeon + AMX
- ktransformers:2026 年 5 月 main 分支
### Pull Request
_No response_
Contributor guide
Research direction
Start by tracing kt_ep_wrapper initialization of gpu_experts_mask and how it receives SGLang's tensor- and expert-parallel ranks. Reproduce the model-loading failure with TP=4 and EP=2, then verify that local expert counts and --kt-num-gpu-experts match each EP rank instead of the full expert count.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100