[Bug] CacheBlend non-prefix KV cache reuse broken with vLLM V1 connector

Summary

CacheBlend's non-prefix KV cache reuse does not work with the vLLM V1 connector (LMCacheConnectorV1Impl). The scheduler-side lookup (get_num_new_matched_tokens) only performs prefix matching, so only the common prefix tokens are marked as "hit." The worker-side blending logic only processes tokens already marked as hits by the scheduler, resulting in no non-prefix cache reuse — the core feature CacheBlend is designed to provide.

Environment

• LMCache: upstream dev branch (commit f35456e1)
• vLLM: 0.18.0+rocm700 (also reproduced on 0.17.1+rocm700)
• GPU: AMD Instinct MI300X (ROCm 7.0) — but this is NOT a ROCm-specific issue
• Model: Qwen2.5-1.5B-Instruct
• Attention backend: LMCTritonSparseBackend (PR #3092), also tested with default

Reproduction

Using the blog post's prompt structure ([system_prompt] + SEP + [user_question] + SEP + [document] + SEP), where the document (~2400 tokens) is identical across two turns but the question changes:

# Turn 1: sys + sep + Q1 + sep + document + sep
# Turn 2: sys + sep + Q2 + sep + document + sep  (same document, different question)

Expected: CacheBlend reuses the document's KV cache from Turn 1 in Turn 2, achieving ~90%+ cache hit rate (as shown in the blog post).

Actual: Only 24/2451 tokens hit (the common system prompt prefix). The document segment is NOT reused.

Log evidence

Turn 1 stores correctly:

Reqid: 1-a5d6eae5, Total tokens 2453, LMCache hit tokens: 0, need to load: 0
[req_id=1-a5d6eae5] Stored 2300 out of total 2304 tokens

Turn 2 only matches prefix:

Reqid: 2-a5cd7733, Total tokens 2451, LMCache hit tokens: 24, need to load: 24
[req_id=unspecified] Retrieved 24 out of 24 out of total 24 tokens

The 24 hit tokens correspond exactly to the system prompt length — the document chunk (which has identical content across both turns) is not recognized.

Root Cause Analysis

The issue is in the two-phase connector architecture in vLLM V1:

1. Scheduler phase (get_num_new_matched_tokens): Uses TokenDatabase.process_tokens() which computes rolling hash chains. Each chunk's hash depends on the previous chunk's hash. Since the question segment (segment 2) differs between turns, all subsequent segment hashes (including the document) differ — even though the document content is identical.

2. Worker phase (start_load_kv → blend()): The blending logic in LMCBlender only processes tokens marked as "hit" by the scheduler. Since the scheduler only found 24 prefix hits, the blender never gets the opportunity to reuse the document's KV cache.

In other words, SegmentTokenDatabase correctly splits the prompt by separator tokens, but the hash chain is still positional — each segment's hash depends on all preceding segments. CacheBlend's content-based matching (which should recognize identical document content regardless of position) appears to only operate at the worker level, but the scheduler has already decided these tokens aren't cached.

Additional Issues Found

While reproducing, we also encountered:

1. ZeroDivisionError in memory_management.py:1134: When store_layer encounters a 0-token chunk (e.g., from separator boundaries), get_size_bytes() returns 0, causing aligned_size=0 and division by zero in batched_allocate. Fixed with a if num_tokens == 0: continue guard in store_layer.

2. CUDAGraphWrapper not unwrapped (vLLM 0.18 + CUDAGraphs): VLLMModelTracker.register_model() registers the CUDAGraphWrapper instead of the underlying model. infer_model_from_vllm() then fails with NotImplementedError: Model type CUDAGraphWrapper is not supported. Fix: unwrap via vllm_model.runnable before type checking.

3. [1:] BOS-skip assumption in SegmentTokenDatabase.__init__: The separator tokenization uses tokenizer.encode(blend_special_str)[1:] which assumes a BOS token. Models without BOS (e.g., Qwen2.5) lose a real content token, producing a 1-token separator that's too common. Should use add_special_tokens=False instead.

4. Double-unpin warnings: Pin count of MemoryObj is negative: -1 occurs consistently during blend retrieval, suggesting a reference counting bug in the memory allocator.

Expected Behavior

The scheduler-side lookup should support non-prefix matching for CacheBlend-enabled sessions, either by:

• Using content-based hashing (independent of position) for segments after a separator boundary
• Or deferring the non-prefix hit detection to the worker side and feeding it back

References

• Blog post: https://blog.lmcache.ai/en/2026/04/01/accelerating-openclaw-agents-with-cacheblend/
• CacheBlend paper: https://arxiv.org/abs/2405.16444
• PR #3092: ROCm Triton sparse attention backend (merged)

Trace get_num_new_matched_tokens through TokenDatabase.process_tokens() and SegmentTokenDatabase to understand how separator segments are hashed. Then inspect LMCBlender and the start_load_kv path, along with memory_management.py:1134 and the listed wrapper and BOS-handling issues. Done means the reproduced two-turn prompt reuses the identical document segment rather than reporting only the 24-token prefix hit.