mudler / mudler/vllm.cpp

port(KV-OFFLOAD): the offload identity is never parallelism-agnostic, so a cache cannot be reread under a different TP degree

Open
#2,639 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: KV-OFFLOAD

Found by re-deriving upstream a8f296083f (vllm#49858) against this tree in
#2632, wave PORTQ-1.

The gap

FileMapper and the offload cache identity both exist here
(include/vllm/v1/kv_offload/fs_io.h:86,
include/vllm/v1/kv_offload/cache_identity.h), but the mechanism the commit
widens does not. CacheIdentity carries tp_size / pp_size / pcp_size /
dcp_size / rank unconditionally in its digest
(include/vllm/v1/kv_offload/cache_identity.h:139-143), and the on-disk path
always embeds the rank —
include/vllm/v1/kv_offload/fs_io.h:97, <base>_r<rank>/<hhh>/<hh>_g<group>/<hash>.bin.
parallel_agnostic appears only in an explanatory comment at
cache_identity.h:46-47.

So a cache written under one TP degree can never be read under another, whatever
the layout.

Already recorded

.agents/specs/kv-persistence-lmcache.md:133 is feature row 50 and already
carries this: "parallel_agnostic folder collapsing … file_mapper.py:46-48,85-96
MISSING". This issue gives that row an owner and a tracked number; it does
not re-discover it.

Upstream

file_mapper.py @ a8f296083f (a forward reference past the pin 5559679229):
:38,64-67 add a replicated_layout constructor argument written into the hashed
fields dict only when true, so existing paths keep their identity, and
:100-104 widen parallel_agnostic to
parallel.is_parallelism_agnostic or config.replicated_layout.

Size

The commit's own delta is ~10 lines: one optional identity field plus one
predicate widening. It is unreachable until the prerequisite lands.
parallel_agnostic gating, with the MLA exclusion and its four cases, is roughly
60-120 lines plus tests; upstream's tests/v1/kv_offload/test_file_mapper.py is
the executable specification, already cited at
.agents/specs/kv-persistence-lmcache.md:407.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

After the prerequisite parallel-agnostic mechanism lands, start with include/vllm/v1/kv_offload/cache_identity.h and fs_io.h, then compare the upstream file_mapper.py change at a8f296083f. Use tests/v1/kv_offload/test_file_mapper.py as the executable specification, including the MLA exclusion and four cases. Done means replicated layouts preserve existing identities while enabling the intended parallel-agnostic cache paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.