mudler / mudler/vllm.cpp

port(KV): the KV-cache layout refactor stages 4-6 are one unit, and all three are inside the current pin target

Open
#2,695 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: none exists. No row in .agents/roadmap_v1.md or any *-matrix.md owns the KV-cache layout refactor series. The nearest existing rows are KV-DSV4-MULTICACHE (.agents/engine-matrix.md:113), which owns the MLA page-size constants this touches but not AttentionSpec, and KV-HYBRID-COORD (:100). A series row is warranted; this issue does not invent one.

Found by wave PORTQ-5 (#2679) re-deriving PORT-NOW entry [167], upstream 57bd0ed441 vllm#51704, stage 5 of the series. Nothing was executed.

What is portable today

Upstream hoists head_size_v from the subclasses onto AttentionSpec; adds backend-published num_head_slots / state_content_bytes and the derived num_heads / state_content_size_bytes; collapses five per-subclass real_page_size_bytes overrides into one unpadded_page_size_bytes = num_heads * storage_block_size * state_content_size_bytes; and adds the AttentionBackend.customize_spec(spec) -> spec hook.

Against this tree:

  • head_size_v is still per-subclass — include/vllm/v1/kv_cache_interface.h:139-157 (base members stop at indexes_kv_by_block_stride), :239 (Full), :352 (SlidingWindow).
  • Five separate real_page_size_bytes() overrides carry the page-size math: src/vllm/v1/kv_cache_interface.cpp:83-91, :107-117, :122-153 (MLA, with the 584/656 fp8_ds_mla constants), :158-181, :183-193. No unpadded_page_size_bytes function exists; the only occurrence of that string is a prose comment at :173.
  • AttentionBackend already carries classmethod-shaped virtual hooks (include/vllm/v1/attention/backend.h:286-300, :356-365), so customize_spec has a home.

What is NOT portable

The TQFullAttentionSpec deletion and the four turboquant KVQuantMode members are surface-absent. KVQuantMode here stops at kNvfp4 = 5 (include/vllm/v1/kv_cache_interface.h:108-115) — which matches the pin exactly — and include/vllm/v1/kv_cache_dtype.h:86-89 refuses turboquant_* outright. Flipping them would land on nothing.

The scheduling constraint, which is the point of this issue

.agents/sync/2026-09-02-db92053.md:127-144 records it: stages 4 (61874f9842) and 5 (this one) are inside 5559679229..e126687a9a, and stage 6 (8bdc70ec7b) is not. Porting 4 and 5 without 6 lands a half-migrated layout surface that no upstream revision ever had.

So this entry must not land alone. It is a three-stage unit preceded by two preparatory steps the sync records already name:

  1. Bring unify_hybrid_kv_cache_specs (src/vllm/v1/core/kv_cache_utils.cpp:26-93) up to pin state (db92053:1876-1887).
  2. Convert the positional spec constructors to designated initializers before any stage — head_size_v, sliding_window and attention_chunk_size are all int, so the hoist silently re-binds positional call sites across ~8 model registries (cdefd9d:1867-1875).

Then either the pin target moves to 8bdc70ec7b or later, or the wave stops before stage 4.

Reachability

customize_spec needs a production call site as well as a virtual. Today the spec is built by model registries (e.g. src/vllm/model_executor/models/qwen3_5_common.cpp:72-90) which never consult a backend. Without that call site the hook lands dead, per AGENTS.md §"Nothing lands dead".

Size

~250-400 lines across include/vllm/v1/kv_cache_interface.h, src/vllm/v1/kv_cache_interface.cpp, include/vllm/v1/attention/backend.h, the ~8 registry call sites and tests/vllm/v1/test_kv_cache_interface*. Larger than the "4 core + 2 test files" .agents/sync/2026-09-01-cdefd9d.md:1846 estimates, because the constructor conversion is not in that count.

Open question, carried forward not resolved

cdefd9d:1901-1907 asks whether state_content_bytes is the right home for kFp8DsMlaV4TokenBytes / kFp8DsMlaV32TokenBytes. Those constants are at src/vllm/v1/kv_cache_interface.cpp:135,139,164 and are still selected by cache_dtype_str rather than by a backend hook, so the question is still open. It needs reading how vllm/models/deepseek_v4/attention.py publishes those bytes after stage 5.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .agents/sync/2026-09-02-db92053.md:127-144 and the preparatory changes in src/vllm/v1/core/kv_cache_utils.cpp and the model registry constructors. Read the listed kv_cache_interface, attention/backend, and test files, then inspect deepseek_v4/attention.py for state-content bytes. Done means stages 4-6 land together at the required pin, customize_spec has a production call site, and the KV-cache tests cover the migrated layout.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
ai-infra-agents, backend, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.