mudler / mudler/vllm.cpp

fix(KV-MANAGER-ALLOC): a short external cache hit requests a negative block count, inflating the free-block counter before it throws

Open
#2,733 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: KV-MANAGER-ALLOC (.agents/engine-matrix.md:98, "Slot allocation, watermark, admission, release"), whose recorded upstream test anchor is tests/v1/core/test_single_type_kv_cache_manager.py — the very file this commit extends. Resolved by line number before filing.

Found by wave PORTQ-7 (#2717) re-deriving PORT-NOW entry [266], upstream 53bf990502 vllm#52707. Nothing was executed.

The defect, and why it is worse here than upstream

SingleTypeKVCacheManager::allocate_external_computed_blocks (src/vllm/v1/core/single_type_kv_cache_manager.cpp:132-158) carries upstream's unclamped expression verbatim:

std::vector<KVCacheBlock*> allocated_blocks = block_pool.get_new_blocks(
    cdiv(num_total_computed_tokens, block_size) -
    static_cast<int>(req_blocks.size()));                     // :148-150

When an external or connector hit is shorter than the blocks the request already holds, the count goes negative. Upstream's Python then merely returned []. This tree does something worse, and each link was read:

  1. BlockPool::get_new_blocks (src/vllm/v1/core/block_pool.cpp:340-345) tests num_blocks > get_num_free_blocks(), which a negative count passes, and forwards to popleft_n.
  2. FreeKVCacheBlockQueue::popleft_n (src/vllm/v1/core/kv_cache_utils.cpp:781-791) skips its if (n == 0) return {}; guard, passes assert(num_free_blocks >= n) trivially — and that assert is compiled out under NDEBUG in any case — then executes num_free_blocks -= n, inflating the free-block counter. That is precisely the pool corruption upstream's new regression test asserts against.
  3. It then calls ret.reserve(static_cast<size_t>(n)) with a negative int cast to size_t, which throws std::length_error. The pop loop never runs, so the counter stays inflated behind the throw.

There is no override to soften it: allocate_external_computed_blocks appears exactly twice in this tree — the declaration at include/vllm/v1/core/single_type_kv_cache_manager.h:123 and this definition. No subclass overrides it.

Reachability

src/vllm/v1/core/kv_cache_coordinator.cpp:213-216 -> src/vllm/v1/core/kv_cache_manager.cpp:268-273 -> the connector's num_external_computed_tokens at src/vllm/v1/core/sched/scheduler.cpp:894, live whenever --kv-transfer-config selects a connector.

Honest limit on that claim. No configuration in this tree was found that actually drives the count negative through KVCacheManager::allocate_slots, because here allocate_new_computed_blocks runs before allocate_new_blocks (kv_cache_manager.cpp:268-280) and early-returns for any already-tracked request (kv_cache_coordinator.cpp:191-205), so the block table is empty on entry. Upstream's own regression test likewise drives the manager directly rather than through allocate_slots. What would settle it: enumerating the get_num_skipped_tokens overrides — SlidingWindowManager (:536), ChunkedLocalAttentionManager (:633), MambaManager (:887, which returns n-1) — against a multi-group config with a live connector.

The clamp is worth porting regardless: it is one line guarding a counter corruption plus a throw, and it costs nothing if the path is unreachable.

The commit is the whole distance

git show 5559679229:vllm/v1/core/single_type_kv_cache_manager.py:282-319 is exactly the ported shape. Not a pre-pin hole. No later in-range commit touches the clamp.

Size

2-3 product lines — the std::max(0, ...) clamp before get_new_blocks — plus one ported regression test. Upstream's test_external_computed_blocks_do_not_corrupt_free_pool is ~40 lines and asserts that get_num_free_blocks() is unchanged and the block table stays at 3; port it with its parameters and its assertions rather than re-deriving.

Consider also clamping or refusing inside popleft_n. A negative n reaching that function is a caller bug in every case, and it currently corrupts state before it throws. That is a second, independent hardening this issue does not require but a reviewer should weigh.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with allocate_external_computed_blocks in src/vllm/v1/core/single_type_kv_cache_manager.cpp and compare its behavior with tests/v1/core/test_single_type_kv_cache_manager.py. Port the upstream regression test for a short external cache hit and verify the free-block count is unchanged and the block table remains at three blocks; the clamp belongs immediately before get_new_blocks.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.