[Bug][GPU Connector] Silent KV cache corruption on vLLM 0.26 fused/packed KV layout

Nobody has claimed this yet.

**Label**

`bug`

**Describe the bug**

On vLLM 0.26, the in-process connector (`LMCacheConnectorV1`) **silently returns wrong
KV data** for models whose KV cache uses vLLM's fused/packed ("unified") layout, where
K and V share the trailing per-head dimension. A request served from the LMCache CPU
tier produces different output than the same request computed from scratch.

There is no exception and no warning — generations are simply wrong.

Setting `use_gpu_connector_v3=True` is not a workaround; it fails differently, with a
hard crash on the first store:

```
File "lmcache/v1/cache_engine.py", line 558, in store
self.gpu_connector.batched_from_gpu(memory_objs, starts, ends, **kwargs)
File "lmcache/v1/gpu_connector/gpu_connectors.py", line 619, in from_gpu
memory_obj_tensor.copy_(tmp_gpu_buffer, non_blocking=True)
RuntimeError: The size of tensor a (1024) must match the size of tensor b (2048) at
non-singleton dimension 3
```

So on this stack both in-process code paths are unusable. `LMCacheMPConnector` is
**not** affected and works as a workaround.

Severity: silent data corruption in the default configuration.

**To Reproduce**

Save as `repro_lmcache_fused_kv.py`. Requires only vLLM + LMCache — no dataset, no
server (except for the optional `lmcache-mp` control). Runs in ~2 minutes per mode and
exits non-zero when the bug is present.

vLLM's own prefix cache is disabled, so the second identical request **must** be served
by the offload tier. Under greedy decoding both generations must be byte-identical.

The script also **asserts that a transfer actually happened** before comparing outputs,
by hooking the exact call each connector makes to pull KV back from its CPU tier — so a
`MATCH: True` cannot be a silent no-op, and a `lmcache` failure cannot be "the cache was
skipped". It ships with three controls (`baseline`, `vllm-offload`, `lmcache-mp`) which
are what make a failure attributable.

```python
#!/usr/bin/env python3
"""Repro: LMCache in-process connector corrupts KV on vLLM's fused/packed KV layout.

python repro_lmcache_fused_kv.py lmcache # BUG: MATCH False
python repro_lmcache_fused_kv.py baseline # control: MATCH True
python repro_lmcache_fused_kv.py vllm-offload # control: MATCH True
python repro_lmcache_fused_kv.py lmcache-mp # control: MATCH True (needs MP server)

vLLM's own prefix cache is disabled, so the 2nd identical request cannot be served
by vLLM and must come from the offload tier. Greedy decoding => both generations
must be byte-identical.

The script also ASSERTS that a transfer really happened, so "MATCH: True" cannot be
a silent no-op (see check_offload_happened).
"""

import os
import sys

MODES = ("lmcache", "lmcache-mp", "vllm-offload", "baseline")
MODE = sys.argv[1] if len(sys.argv) > 1 else "lmcache"
if MODE not in MODES:
sys.exit(f"unknown mode {MODE!r}; choose one of {MODES}")

os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0" # engine in-process
if MODE.startswith("lmcache"):
os.environ["LMCACHE_CHUNK_SIZE"] = "256"
os.environ["LMCACHE_LOCAL_CPU"] = "True"
os.environ["LMCACHE_MAX_LOCAL_CPU_SIZE"] = "8"
os.environ.setdefault("PYTHONHASHSEED", "0") # required by the MP connector

from vllm import LLM, SamplingParams # noqa: E402
from vllm.config import KVTransferConfig # noqa: E402

ktc = None
if MODE == "lmcache":
ktc = KVTransferConfig(kv_connector="LMCacheConnectorV1", kv_role="kv_both")
elif MODE == "lmcache-mp":
ktc = KVTransferConfig(
kv_connector="LMCacheMPConnector", kv_role="kv_both",
kv_connector_extra_config={"lmcache.mp.host": "tcp://127.0.0.1",
"lmcache.mp.port": 6555})
elif MODE == "vllm-offload":
ktc = KVTransferConfig(
kv_connector="OffloadingConnector", kv_role="kv_both",
kv_connector_extra_config={"cpu_bytes_to_use": 8 << 30, "block_size": 256})

# ---------------------------------------------------------------------------
# Offload verification hooks.
#
# Each hook wraps the one call the connector makes to actually pull KV back from
# its CPU tier. Counting those calls proves the 2nd request was SERVED FROM CACHE
# rather than silently recomputed -- without this, a no-op connector would "pass".
# ---------------------------------------------------------------------------
LOADS: list[int] = []

if MODE == "lmcache":
from lmcache.v1.gpu_connector import gpu_connectors as _gc

for _cls in ("VLLMPagedMemGPUConnectorV2", "VLLMPagedMemGPUConnectorV3"):
_k = getattr(_gc, _cls, None)
if _k is None:
continue

def _wrap(klass):
_orig = klass.to_gpu

def _counting_to_gpu(self, memory_obj, start, end, **kw):
LOADS.append(memory_obj.get_size())
return _orig(self, memory_obj, start, end, **kw)

klass.to_gpu = _counting_to_gpu

_wrap(_k)

elif MODE == "lmcache-mp":
from lmcache.integration.vllm import vllm_multi_process_adapter as _mpa

_MPW = _mpa.LMCacheMPWorkerAdapter
_orig_submit = _MPW.batched_submit_retrieve_requests

def _counting_submit(self, request_ids, ops, event, **kw):
LOADS.append(len(request_ids))
return _orig_submit(self, request_ids, ops, event, **kw)

_MPW.batched_submit_retrieve_requests = _counting_submit

elif MODE == "vllm-offload":
from vllm.v1.kv_offload.cpu.manager import CPUOffloadingManager

_orig_prepare_load = CPUOffloadingManager.prepare_load

def _counting_prepare_load(self, keys, req_context):
LOADS.append(len(list(keys)))
return _orig_prepare_load(self, keys, req_context)

CPUOffloadingManager.prepare_load = _counting_prepare_load

def check_offload_happened() -> str:
"""Prove request #2 was served from the offload tier rather than recomputed.

Returns:
A human-readable evidence string.

Raises:
SystemExit: If no KV was actually loaded from the offload tier.
"""
if MODE == "baseline":
return "n/a - baseline has no offload tier by design"
if not LOADS:
sys.exit(
f"VERIFY FAILED [{MODE}]: the offload tier was never asked to load. "
"vLLM served the request itself, so this run proves nothing. Check "
"that enable_prefix_caching=False."
)
if MODE == "lmcache":
return f"LMCache loaded {sum(LOADS) / 1e6:.1f} MB in {len(LOADS)} chunks"
if MODE == "lmcache-mp":
return f"LMCache MP issued {sum(LOADS)} retrieve request(s)"
return f"vLLM loaded {sum(LOADS)} chunks from its CPU tier"

llm = LLM(model="Qwen/Qwen3-8B", kv_transfer_config=ktc, max_model_len=8192,
gpu_memory_utilization=0.85,
# MUST stay False. With prefix caching on, vLLM's GPU cache serves the
# 2nd request itself, the offload tier is never consulted and the bug is
# masked -- verified: "need to load: 0", MATCH True on an affected build.
enable_prefix_caching=False)

# See "prompt sensitivity" note in the issue -- deliberately semi-repetitive.
prompt = ("Reference dossier 4815162342.\n\n"
+ "The quick brown fox jumps over the lazy dog near the riverbank. " * 260
+ "\n\nQuestion: Summarize the passage in one sentence.\nAnswer:")

sp = SamplingParams(max_tokens=32, temperature=0.0) # greedy => deterministic
first = llm.generate([prompt], sp)[0].outputs[0].text # full compute
LOADS.clear() # only inspect request #2
second = llm.generate([prompt], sp)[0].outputs[0].text # served from the KV cache

evidence = check_offload_happened()

print(f"\nmode: {MODE}")
print(f"offload verified: {evidence}")
print(f"1st (compute): {first!r}")
print(f"2nd (cached) : {second!r}")
print(f"MATCH: {first == second}")
os._exit(0 if first == second else 1)
```

Run:

```bash
python repro_lmcache_fused_kv.py lmcache # exit 1 <-- the bug
python repro_lmcache_fused_kv.py baseline # exit 0 (control)
python repro_lmcache_fused_kv.py vllm-offload # exit 0 (control)
python repro_lmcache_fused_kv.py lmcache-mp # exit 0 (control, needs MP server)
```

The `lmcache-mp` control needs a server:

```bash
python -m lmcache.v1.multiprocess.server --host 0.0.0.0 --port 6555 \
--l1-size-gb 20 --eviction-policy LRU --max-workers 4 --chunk-size 256
```

All four modes on the affected build:

```
### lmcache exit=1 offload verified: LMCache loaded 528.5 MB in 14 chunks MATCH: False
### vllm-offload exit=0 offload verified: vLLM loaded 14 chunks from its CPU tier MATCH: True
### lmcache-mp exit=0 offload verified: LMCache MP issued 1 retrieve request(s) MATCH: True
### baseline exit=0 offload verified: n/a - baseline has no offload tier MATCH: True
```

**Two things that will stop you reproducing it**

1. **`enable_prefix_caching` must stay `False`.** With it on, vLLM's own GPU prefix
cache serves the second request, the offload tier is never consulted, and the bug
is masked (verified: `need to load: 0`, `MATCH: True` on an affected build). The
bug does still occur with prefix caching on, but only once the GPU cache evicts,
which needs many documents and a capped KV cache — a far more fragile repro.
2. **The prompt matters.** Too repetitive and the bug is masked (a false negative we
hit); too random and greedy output is degenerate and flips on any numerical noise
(`vllm-offload`, a known-correct connector, also reported `MATCH: False`). The
prompt above was validated to discriminate across all four modes. **If a control
fails, distrust the prompt, not the connector.**

Observed:

```
mode: lmcache
1st (compute): ' The passage consists of multiple repetitions of the sentence "The quick brown
fox jumps over the lazy dog near the riverbank." ...'
2nd (cached) : ' The passage is a series of the same sentence about a quick brown fox jumps
over the lazy dog near the riverbank." \n\nQuestion: What is the main idea'
MATCH: False
```

Deterministic — identical wrong output across runs. All three controls pass, so
decoding determinism and the harness itself are sound.

**Expected behavior**

`MATCH: True`. A cache hit must be numerically identical to a full recompute.

**Environment**

| | |
|---|---|
| LMCache | `dev` @ `8932ed07` |
| vLLM | 0.26.0 |
| Model | Qwen/Qwen3-8B (36 layers, 8 KV heads, head_dim 128, bf16) |
| Hardware | 1x H100 80GB, TP=1 |
| OS / Python | Linux, Python 3.12, torch 2.11.0+cu130 |
| Config | `chunk_size=256`, `local_cpu=True`, non-layerwise, `use_gpu_connector_v3=False` (default) |

Run repro_lmcache_fused_kv.py in the four listed modes and confirm that only lmcache fails while the controls match. Then inspect lmcache/v1/gpu_connector/gpu_connectors.py around the from_gpu and to_gpu paths, plus lmcache/v1/cache_engine.py around store. Done means cached and full-compute generations match without the reported tensor-size crash.

Written by the indexing model from the issue text.