[Feature][MP] Commit windowed KV to L2 on turn boundaries
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 11.9k
- Forks
- 1.9k
- Avg merge
- 4d 4h
- Merged PRs (30d)
- 141
Description
# Problem
Hybrid models (Gemma 3, gpt-oss, Command-A, Qwen3-Next, ...) have object groups that only need the last `w` chunks for a prefix hit (SWA, Mamba). Older windowed chunks are never read back, and they are the big ones. `default` writes them all to L2. But skipping them isn't enough: once L1 evicts the window, the next turn pays a full prefill. We should commit only the last w chunks, at the turn boundary.
# Proposal
Two pieces that work together: stop writing windowed KV to L2, and copy the live window to L2 at natural timing (turn boundaries) so cross-turn prefix reuse isn't lost.
## 1. defer_windowed store policy
A new store policy that writes full-attention object groups through to L2 (same as default) but keeps windowed groups L1-only, eliminating the dead-weight writes. This policy requires `--separate-object-groups`.
## 2. window commit on turn boundaries
When a chat turn ends (vLLM sends END_SESSION), copy the live windowed layer’s window from L1 to L2. This is a copy, not a move: the L1 window stays for immediate reuse.
## Key design points:
- `CommitPolicy` decides per-request whether to commit, based on how the request ended. The built-in `turn_end` policy commits when the generation ended on a turn-end token (configurable via `--turn-boundary-token–ids`) at generation_end, and for any finish reason other than abort / error at prompt_end, where the prompt is re-sent regardless.
- `--window-commit-anchor (generation_end | prompt_end)` controls where the window ends. Normally the next prompt re-sends the previous assistant turn verbatim, so `generation_end` (default) — right when generation stops — is the correct anchor. `prompt_end` covers cases where the client re-renders the assistant turn differently before resending it (e.g. Qwen3 drops `` blocks). It anchors the window at the end of the looked-up prompt.
- `StoreController.submit_copy` copies keys to L2 without consulting the store policy and without evicting them from L1.
Extensible: custom policies register with `register_commit_policy_factory`.
I will send the draft implementation PR later. I work for this issue with @nilpe and @Alnusjaponica .
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with CommitPolicy, StoreController.submit_copy, and the register_commit_policy_factory entry point; trace how END_SESSION and generation_end/prompt_end handling are represented. Done means defer_windowed avoids windowed L2 writes while turn_end commits the live window at the configured boundary without evicting L1 data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100