kvcache-ai / kvcache-ai/ktransformers
[Bug] Kimi K2 Thinking decode produces invalid tokens when using kt-kernel / kt-amx weights with cpu experts
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Checklist
- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.
### Describe the bug
After installing kt-kernel and using the command given in the readme, though the server starts up and responds to requests, the content of the K2-Thinking is clearly broken, just repeating tokens over and over.
I tried to change a few sizing options, and while it successfully is "decoding" the tokens its decoding don't seem to be right at all.
Not sure if this is a sglang or ktransformers bug though.
### Reproduction
```bash
$ python -m sglang.launch_server --model moonshotai/Kimi-K2-Thinking --kt-amx-weight-path /path/to/home/.cache/huggingface/hub/models--KVCache-ai--Kimi-K2-Thinking-CPU-weight/snapshots/690ffacb9203d3b5e05ee8167ff1f5d4ae027c83 --kt-cpuinfer 128 --kt-threadpool-count 1 --kt-num-gpu-experts 200 --kt-amx-method AMXINT4 --trust-remote-code --mem-fraction-static 0.98 --chunked-prefill-size 4096 --max-running-requests 10 --max-total-tokens 37000 --enable-mixed-chunk --tensor-parallel-size 4 --enable-p2p-check --disable-shared-experts-fusion
```
To test, ran:
```bash
$ curl -N http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -H "Accept: text/event-stream" -d '{
"model": "moonshotai/Kimi-K2-Thinking",
"messages": [{"role": "user", "content": "Reply with exactly: hello"}],
"temperature": 0,
"stream": true
}'
```
Result (note the output is just repeating tokens "assistant assistant assistant" etc.)
```bash
data: {"id":"40204d5b7303457783dcb39f3e136ec3","object":"chat.completion.chunk","created":1762466824,"model":"moonshotai/Kimi-K2-Thinking","choices":[{"index":0,"delta":{"role":"assistant","content":"","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}
data: {"id":"40204d5b7303457783dcb39f3e136ec3","object":"chat.completion.chunk","created":1762466824,"model":"moonshotai/Kimi-K2-Thinking","choices":[{"index":0,"delta":{"role":null,"content":" actually","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}
data: {"id":"40204d5b7303457783dcb39f3e136ec3","object":"chat.completion.chunk","created":1762466824,"model":"moonshotai/Kimi-K2-Thinking","choices":[{"index":0,"delta":{"role":null,"content":" assistant","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}
data: {"id":"40204d5b7303457783dcb39f3e136ec3","object":"chat.completion.chunk","created":1762466824,"model":"moonshotai/Kimi-K2-Thinking","choices":[{"index":0,"delta":{"role":null,"content":" assistant","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}
data: {"id":"40204d5b7303457783dcb39f3e136ec3","object":"chat.completion.chunk","created":1762466824,"model":"moonshotai/Kimi-K2-Thinking","choices":[{"index":0,"delta":{"role":null,"content":" assistant","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}
data: {"id":"40204d5b7303457783dcb39f3e136ec3","object":"chat.completion.chunk","created":1762466824,"model":"moonshotai/Kimi-K2-Thinking","choices":[{"index":0,"delta":{"role":null,"content":" t","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}
...
```
### Environment
OS: Ubuntu
GPU: 4xRTX Pro 6000 Blackwell
CPU: AMD Ryzen Threadripper PRO 9995WX 96-Cores
RAM: 1TB
installed kt-kernel and sglang in the same virtualenv
```bash
uv venv
source .venv/bin/activate
uv pip install "sglang[all]"
uv pip install git+https://github.com/kvcache-ai/ktransformers/tree/main/kt-kernel
```
(also, I installed ktransformers as well,
```bash
export CMAKE_ARGS="-DCMAKE_CUDA_ARCHITECTURES=100 -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc"
uv pip install git+https://github.com/kvcache-ai/ktransformers.git
```
Also, for external dependencies i installed:
```bash
sudo apt-get install -y libblis-openmp-dev libhwloc-dev libnuma-dev pkg-config
```
Nvidia information:
```
NVIDIA-SMI 580.95.05 Driver Version: 580.95.05
```
Contributor guide
Research direction
Start by reproducing the README launch command through sglang.launch_server with the kt-kernel/kt-amx weights and CPU experts, then compare the generated output with the exact curl request and environment details reported here. Done means Kimi K2 Thinking produces valid, non-repeating tokens under this configuration; the issue does not identify a source file or test to target.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100