awslabs / awslabs/keys_values

Roll back small number of KV cache updates

Open
#76 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
18
Forks
8
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
I'd like to be able to roll back a small number of KV cache updates, without having to checkpoint the full cache. Arises in different situations:

- Evaluation: Compute different metrics, given the same prompt. Or sample different responses
- Speculative decoding

**Describe the solution you'd like**
This is essentially just a special case of the annotations used in `autograd_hooks.py`. We need to store `index` and `delta` for every update, so that cache buffers can be restored. Ideally, this is supposed by some methods in the `KVCache` base class.

Contributor guide

Open the contributing guide

Research direction

Start by reading the annotation approach in autograd_hooks.py and locating the KVCache base class. Trace how cache updates change the buffers, then determine how index and delta data can support restoring a small set of updates. Done means callers can roll back selected KV cache updates without checkpointing the full cache, including the evaluation and speculative-decoding cases described.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.