kvcache-ai / kvcache-ai/ktransformers

[Bug] New Update breaks Inference for Balance Serve with Cache (gpu_only: false)

Open
#1,461 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

### Describe the bug

Hi,
It looks like the latest version of Ktransformers ( SmallThinker and GLM4-MoE update ) breaks the kvc2 cache. After a successful install, I set the kvc2 config to:

kvc2:
gpu_only: true
utilization_percentage: 1.0
cpu_memory_size_GB: 500
disk_path: /mnt/data/persist-kvs

and every model I load and try to do inference on gives me this error in the rpc.log file:

[2025-07-30 19:35:15.530] [info] [prefix.cpp:1303] No Match, No need to load
[2025-07-30 19:35:15.530] [error] [prefix.cpp:1336] GPU Cache Layer Count not match
python3: /mnt/home_extend/llm/ktransglm/ktransformers/csrc/balance_serve/kvc2/src/prefix.cpp:1337: virtual void kvc2::KVC2::lookup_to_gpu_async(ModelName, QuantType, kvc2::Token*, kvc2::TokenLength, kvc2::TokenLength, std::function): Assertion `false' failed.

with some debug prints, it looks like the "h->k_info().hidden_layer_count()" on line 1335 of prefix.cpp does not properly return the layer count of the model. In my tests it returns 1.

Any ideas what it could be?

### Reproduction

-

### Environment

W7-3455
512GB DDR5
RTX 4090 ( CUDA 12.8 )

Contributor guide

Open the contributing guide

Research direction

Start with csrc/balance_serve/kvc2/src/prefix.cpp around line 1335 and the rpc.log assertion for GPU Cache Layer Count. Reproduce inference using the reported kvc2 configuration, checking the discrepancy between the title's gpu_only setting and the body; done means the model layer count is correct and inference no longer aborts at lookup_to_gpu_async.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.