kvcache-ai / kvcache-ai/ktransformers

[Feature] Support Native K2 Thinking

Open
#1,598 28 comments 2 reactions 1 assignee Claimed by @ouqingliang View on GitHub
enhancement
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

Kimi released [Kimi-K2-Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking), trained by native INT4 QAT method. Kimi-K2-Thinking supports multiple tool calling and stable long-horizon agency. From KTransformers perspective, it is possible for us to support a large model in native precision.

We decided to support this in 2 stages:
1. Support CPU INT4 decode with AVX2 instructions. @ouqingliang
2. Support GPU prefill with layerwise prefill. @chenht2022

We expected achieving a performance of 200~300 prefill tps and ~15 decode tps, to boost local agent application.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.