kvcache-ai / kvcache-ai/ktransformers
[Feature] Insert generated response in kv cache
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Checklist
- [x] 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [x] 2. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-English/Chinese content without translation may be closed.
### Motivation
I noticed when I am generating responses for a single user chat (alternating user and ai messages) then after the ai has generated a message and I send a new prompt following that, it has to re-process the last ai message as if it was a new input. This is unnecessary, it can be inserted into the cache during generation.
### Related resources
_No response_
Contributor guide
Research direction
The issue names no files, tests, or entry points. Trace the response-generation path and KV-cache handling first; done means a generated response is inserted into the cache so the next prompt does not re-process that response.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100