[Bug] Cross-request KV contamination with local disk backend: returning sessions answer with OTHER requests' content (v0.5.2, vLLM 0.26.0)

Hi team,

First off, thanks for maintaining LMCache!

I ran into a critical silent KV cache corruption issue where long-context sessions served from the disk tier end up getting KV cache belonging to other requests.

Below are the details and reproduction steps.

---

Bug Description

When using local_cpu + local_disk backends, long-context sessions returning after their KV cache has been paged to the disk tier receive KV data belonging to other requests.

As a result, the model answers questions using facts from a completely different conversation (and sometimes includes fragments from unrelated concurrent short requests). No errors/warnings are logged—the corruption is entirely silent.

Environment

• LMCache Version: 0.5.2 (latest)
• vLLM Version: 0.26.0 (matched per v0.5.2 release notes)
• GPU: Single GPU (24 GB)
• OS: Linux
• Model: hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 (FP16 KV)
• Config:
  • chunk_size: 256
  • local_cpu: true, max_local_cpu_size: 8
  • local_disk: file://..., max_local_disk_size: 28 (sized large enough so eviction is never triggered, keeping issue #1981 out of scope)
• Connector: {"kv_connector": "LMCacheConnectorV1", "kv_role": "kv_both"}

Steps to Reproduce

1. Created 6 distinct sessions of ~32K tokens each. Each session contains a unique needle ("the vault code is ZK-600<i>") inserted at 10% depth (Needle-In-A-Haystack setup).
   • Total working set ≈ 24 GB KV, forcing most chunks onto the disk tier (local_cpu is limited to 8 GB).
2. Ran a 150-second mixed load:
   • The 6 long-context sessions periodically return (full prompt resent, prefix served via LMCache).
   • Concurrently, 3 background workers send short unrelated requests ("the ticket number is TK-<k>...").
3. Verified the generated answers against each session's ground-truth needle.

Observed Behavior

Returning sessions consistently answered with the WRONG session's needle.

A single-return probe after the load test yielded:

• Session 0: Expected ZK-6000 $\rightarrow$ Got "ZK-6001. The ticket number is TK-"
• Session 2: Expected ZK-6002 $\rightarrow$ Got "ZK-6000. ..."
• Sessions 3, 4, 5: Expected ZK-6003/4/5 $\rightarrow$ All got "ZK-6001 ..."

Notice that responses also contain text fragments from concurrent short requests ("The ticket number is TK-"), confirming KV cross-talk across completely independent requests.

Across the entire 150s window (138 total session returns), only 14 answered with their own needle (~10% accuracy).

Sanity Checks / Controls

• Clean Environment: Successfully reproduced twice on a completely reset environment (fresh server processes, wiped store directories, 40GB free RAM, empty GPU memory).
• Control Test: Running the exact same benchmark harness on stock vLLM (no connector) and another KVConnector implementation yields 100% accuracy, confirming the harness itself is clean.
• Cold Run: Cold runs (initial store population, few returns) do NOT show this bug—it only emerges once requests start being served back from the disk tier at scale.

Expected Behavior

A returning request should only ever be served KV chunks derived from its own exact token prefix. If a key mismatch occurs, it should trigger a cache miss (recompute), never silently inject cross-talk/corrupted KV content.

---

Additional Context

• Note: While isolating this, we also hit the eviction crash already tracked in #1981 (FileNotFoundError in local_disk_backend.remove during batched_remove) on a single-GPU setup, and left a comment there.
• I have the full benchmark script and logs ready if needed. Happy to share them to help debug!

Thanks again for looking into this!

Start by reproducing the issue with the local_cpu and local_disk backends using the six-session mixed-load setup described in the report. Trace how returning requests retrieve disk-tier KV chunks and compare cache keys or prefixes; done means each request receives only its own KV data, with a mismatch producing a cache miss rather than cross-request content.