kvcache-ai / kvcache-ai/ktransformers
[Feature] Support Native K2 Thinking
Open
enhancement
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
Kimi released [Kimi-K2-Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking), trained by native INT4 QAT method. Kimi-K2-Thinking supports multiple tool calling and stable long-horizon agency. From KTransformers perspective, it is possible for us to support a large model in native precision.
We decided to support this in 2 stages:
1. Support CPU INT4 decode with AVX2 instructions. @ouqingliang
2. Support GPU prefill with layerwise prefill. @chenht2022
We expected achieving a performance of 200~300 prefill tps and ~15 decode tps, to boost local agent application.
Contributor guide
Assessment
This issue has not been assessed yet.