kvcache-ai / kvcache-ai/ktransformers

支持 SGLang 的 expert-parallel-size(EP)

Open
#2,002 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Reminder

- [x] I have read the above rules and searched the existing issues.

### Description

目前 kt_ep_wrapper 不支持 SGLang 的 --expert-parallel-size 参数。
TP=8 在 8×L20(无 NVLink)下 PCIe all-reduce 是主要瓶颈,
改用 TP=4+EP=2 通信开销能明显降低。

当前问题

配了 --tensor-parallel-size 4 --expert-parallel-size 2 后,
模型加载阶段崩溃:

RuntimeError: The size of tensor a (192) must match the size of
tensor b (384) at non-singleton dimension 0

gpu_experts_mask 初始化时按 384 个全量 experts 创建,
但 SGLang EP 切分后每个 rank 只剩 192 个 experts,对不上。

期望行为

kt_ep_wrapper 感知 SGLang 的 EP rank,按 EP 切分后的 local expert 数
来创建 gpu_experts_mask,--kt-num-gpu-experts 也应该按 EP rank 自适应。

环境

- 模型:Kimi-K2.6 RAWINT4
- GPU:8×L20(SM89,无 NVLink)
- CPU:双路 Xeon + AMX
- ktransformers:2026 年 5 月 main 分支

### Pull Request

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing kt_ep_wrapper initialization of gpu_experts_mask and how it receives SGLang's tensor- and expert-parallel ranks. Reproduce the model-loading failure with TP=4 and EP=2, then verify that local expert counts and --kt-num-gpu-experts match each EP rank instead of the full expert count.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.