kvcache-ai / kvcache-ai/ktransformers

我用8张L40S跑glm-5,第一个问题速度正常,第二个问题开始就急剧下降到初始速度的四分之一以下。

Open
#1,877 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Reminder

- [x] I have read the above rules and searched the existing issues.

### System Info

没有启用 --fp8-gemm-backend cutlass 因为GPU不支持。
第一个问题,[2026-03-06 00:54:14 TP0] Decode batch, #running-req: 1, #token: 1344, token usage: 0.01, cuda graph: True, gen throughput (token/s): 8.32, #queue-req: 0
第二、三个问题
[2026-03-06 08:45:41 TP0] Decode batch, #running-req: 1, #token: 4480, token usage: 0.02, cuda graph: True, gen throughput (token/s): 1.85, #queue-req: 0
[2026-03-06 09:08:08 TP0] Decode batch, #running-req: 1, #token: 5952, token usage: 0.03, cuda graph: True, gen throughput (token/s): 1.93, #queue-req: 0

conda activate kt
export PYTORCH_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_JIT_DEEPGEMM=0

python -m sglang.launch_server \
--host 0.0.0.0 \
--port 9997 \
--model /media/kkk/dat/GLM-5-FP8 \
--kt-weight-path /media/kkk/dat/GLM-5-FP8 \
--kt-cpuinfer 48 \
--kt-threadpool-count 4 \
--kt-num-gpu-experts 30 \
--kt-method FP8 \
--kt-gpu-prefill-token-threshold 1024 \
--kt-enable-dynamic-expert-update \
--kt-expert-placement-strategy uniform \
--trust-remote-code \
--mem-fraction-static 0.85 \
--served-model-name GLM5 \
--enable-mixed-chunk \
--tensor-parallel-size 8 \
--enable-p2p-check \
--disable-shared-experts-fusion \
--chunked-prefill-size 16384 \
--max-running-requests 4 \
--max-total-tokens 200000 \
--attention-backend flashinfer \
--kv-cache-dtype bf16 \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--watchdog-timeout 3000

### Reproduction

```text
Put your message here.
```

### Others

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with the provided `python -m sglang.launch_server` configuration on eight L40S GPUs, comparing the first and later requests and their throughput logs. Inspect the GLM-5 FP8 inference path and the listed expert, KV-cache, attention, and chunked-prefill settings; done means identifying the cause of the sustained slowdown and documenting or validating a fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.