LMCache / LMCache/LMCache

[Feature][MP] Commit windowed KV to L2 on turn boundaries

Open
#5,073 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
11.9k
Forks
1.9k
Avg merge
4d 4h
Merged PRs (30d)
141

Description

# Problem
Hybrid models (Gemma 3, gpt-oss, Command-A, Qwen3-Next, ...) have object groups that only need the last `w` chunks for a prefix hit (SWA, Mamba). Older windowed chunks are never read back, and they are the big ones. `default` writes them all to L2. But skipping them isn't enough: once L1 evicts the window, the next turn pays a full prefill. We should commit only the last w chunks, at the turn boundary.

# Proposal
Two pieces that work together: stop writing windowed KV to L2, and copy the live window to L2 at natural timing (turn boundaries) so cross-turn prefix reuse isn't lost.

## 1. defer_windowed store policy
A new store policy that writes full-attention object groups through to L2 (same as default) but keeps windowed groups L1-only, eliminating the dead-weight writes. This policy requires `--separate-object-groups`.

## 2. window commit on turn boundaries
When a chat turn ends (vLLM sends END_SESSION), copy the live windowed layer’s window from L1 to L2. This is a copy, not a move: the L1 window stays for immediate reuse.

## Key design points:

- `CommitPolicy` decides per-request whether to commit, based on how the request ended. The built-in `turn_end` policy commits when the generation ended on a turn-end token (configurable via `--turn-boundary-token–ids`) at generation_end, and for any finish reason other than abort / error at prompt_end, where the prompt is re-sent regardless.
- `--window-commit-anchor (generation_end | prompt_end)` controls where the window ends. Normally the next prompt re-sends the previous assistant turn verbatim, so `generation_end` (default) — right when generation stops — is the correct anchor. `prompt_end` covers cases where the client re-renders the assistant turn differently before resending it (e.g. Qwen3 drops `` blocks). It anchors the window at the end of the looked-up prompt.
- `StoreController.submit_copy` copies keys to L2 without consulting the store policy and without evicting them from L1.
Extensible: custom policies register with `register_commit_policy_factory`.

I will send the draft implementation PR later. I work for this issue with @nilpe and @Alnusjaponica .

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with CommitPolicy, StoreController.submit_copy, and the register_commit_policy_factory entry point; trace how END_SESSION and generation_end/prompt_end handling are represented. Done means defer_windowed avoids windowed L2 writes while turn_end commits the live window at the configured boundary without evicting L1 data.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.