Roll back small number of KV cache updates
- Dominant language
- Python
- Stars
- 18
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
I'd like to be able to roll back a small number of KV cache updates, without having to checkpoint the full cache. Arises in different situations:
- Evaluation: Compute different metrics, given the same prompt. Or sample different responses
- Speculative decoding
**Describe the solution you'd like**
This is essentially just a special case of the annotations used in `autograd_hooks.py`. We need to store `index` and `delta` for every update, so that cache buffers can be restored. Ideally, this is supposed by some methods in the `KVCache` base class.
Contributor guide
Research direction
Start by reading the annotation approach in autograd_hooks.py and locating the KVCache base class. Trace how cache updates change the buffers, then determine how index and delta data can support restoring a small set of updates. Done means callers can roll back selected KV cache updates without checkpointing the full cache, including the evaluation and speculative-decoding cases described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100