[Bug]: Incorrect external prefix-cache reuse (out-of-order / off-topic responses) when partial hash hits are enabled for models with large attention blocks (e.g. GLM-5.3-Flash)

### Bug Report

**Target repo:** kvcache-ai/Mooncake
**Related issues:** #3048 (same symptom), #2827 (KV-integrity contamination in SSD offload path)

---

## Related issues (checked before filing)

- **#3048 — [Bug]: 有人遇到过模型答非所问吗** (open, H20 + mooncake as KV offload causing off-topic/wrong answers; reproduced with DeepSeek-V4-Flash). Same symptom as this report, but no root cause was confirmed there — maintainers attributed it to vLLM `400 Bad Request` and it remains open/undiagnosed. This report provides a concrete mechanism (partial hash hits on large-attention-block models).
- **#2827 — [Bug]: [SSD] DSV4 enable SSD offload，stress testing, repeat offloading the same batch key, lead to OBJECT_ALREADY_EXISTS/persist failed/INVALID_KEY** (open). A confirmed KV-integrity bug in the SSD offload path (concurrent duplicate offload → `OBJECT_ALREADY_EXISTS` → whole-batch persist failure → Master/SSD metadata divergence → `INVALID_KEY` on read → cross-request KV contamination).

This issue is filed as a distinct report because, unlike #3048, it identifies a mechanical defect (partial-hash-hit mapping coarse existence onto full-block reuse) with code path and repro; it references #2827 as the related SSD-offload contamination case.

**Checklist**
- [x] vLLM version: `v0.1.dev20051+g487ecf187` (custom build based on official `vllm/vllm-openai:glm53-flash-cu129`)
- [x] Hardware: 8× NVIDIA H20 (cu129), 4×TP per instance
- [x] Mooncake: standalone-store (mc-master + mc-client, 1 TiB shared segment + SSD offload)
- [x] Scenario: two vLLM instances (GPU0-3 / GPU4-7) sharing one mooncake KV segment for cross-instance prefix reuse

---

## Summary

When the Mooncake KV connector enables **partial hash hits** for a model whose **attention block size is much larger than the prefix-hash block size**, in-flight requests can silently reuse a *partial block* from another request. The reused KV block is **not content-identical**, so the model attends over mismatched cached state and produces **out-of-order / off-topic / garbled** responses (observed: a Chinese fault-ticket task returned a long English "reasoning"-style dump that clearly belonged to a different request).

## Environment

- Model: GLM-5.3-Flash0831 (Glm5NextForConditionalGeneration, MLA, MTP speculative 2)
- Serving: `--enable-prefix-caching --max-model-len 200K --gpu-memory-utilization 0.92 --speculative-config '{"method":"mtp","num_speculative_tokens":2}'`
- KV transfer:
```
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_buffer_device":"cuda","kv_buffer_size":1000000000.0,"kv_role":"kv_both"}'
```
- Two instances share one mooncake segment (cross-instance reuse).
- Observed runtime block sizes:
- `Setting kv cache block size to 64` → prefix-hash block size = **64**
- `Setting attention block size to 1152` → attention/KV block size = **1152**
- `Mamba cache mode is set to 'align' for Glm5Next... when prefix caching is enabled`

## Key observation at failure time

`mc-master` admin metrics during the failure window:

```
Mem Storage: 940.90 GB / 1.00 TB (91.9%)
Keys: 127075
Eviction: Success/Attempts=22/1217 # ~98% of evictions FAILED
SSD Storage: 973.60 GB / 2.00 TB
```

Model logs:

```
External prefix cache hit rate: 59-65%
Mooncake load tier summary: batch_keys=36 memory_keys=36 ... success_keys=36 failed_keys=0 (full memory-tier external hit, 526 MB KV reused)
```

A request at the same timestamp returned a multi-KB English reasoning/tooling dump (`gorilla agent mindset`, `127.0.0.1:8644`, `PHP`, `browser repair`, `shopping cart`, `philosophical`, ...) that was unrelated to the actual prompt — classic cross-request KV contamination.

## Root cause analysis (code)

The Mooncake connector mirrors vLLM core's `enable_partial_hash_hits` (`vllm/distributed/kv_transfer/kv_connector/v1/mooncake/store/coordinator.py`):

```python
def partial_hash_hits_enabled(kv_cache_groups, hash_block_size) -> bool:
return any(
isinstance(spec := _unwrap_spec(g.kv_cache_spec), MambaSpec)
and spec.mamba_cache_mode == "align"
and spec.block_size > hash_block_size
for g in kv_cache_groups
)
```

For GLM, `hash_block_size == 64` and the mamba/attention block is aligned to `1152`, so `1152 > 64` → **partial hash hits are enabled**.

The connector's external lookup reports a hit at the **64-token chunk granularity**:

```python
def get_cached_block(self, block_hash, group_ids):
...
h = bytes(block_hash)
if all((g, h) in self._exists for g in group_ids):
return [self._present_block] * len(group_ids) # reuse the whole 1152 block
return None
```

and alignment:

```python
def align_lookup_length(self, length):
alignment = self.hash_block_size if self.enable_partial_hash_hits else self.lcm_block_size
return length // alignment * alignment
```

But the unit that is actually **reused / offloaded is the full 1152-token KV block**. When two requests share a prefix only to a 64-token boundary (but diverge within the same 1152-token block), the connector reports a "hit" and the scheduler reuses a block whose remaining tokens are from a **different request** → mismatched KV → wrong hidden states → garbled / off-topic generation.

In vLLM core, partial-hash handling is safe because the core owns the block pool and can reconstruct/verify the actually-matching sub-blocks. The Mooncake connector's mirror appears to map a coarse chunk existence check onto full-block reuse without that guarantee.

It is aggravated by the shared-store being ~92% full with eviction mostly failing (`attempts=1217, success=22`), which destabilizes the existence mirror and increases spurious/over-long hits, and by high external hit rates (59-65%) across many concurrent requests sharing a long agent system-prompt.

## Expected behavior

External prefix KV reuse should only be reported when the **full KV block** being reused is byte-identical to the current request's tokens at that position. Partial (sub-block) hits must not cause reuse of a mismatched full block.

## Steps to reproduce

1. Serve GLM-5.3-Flash (or any model with large attention block, e.g. 1152, and small hash block, e.g. 64) with:
- `--enable-prefix-caching`
- Mooncake `kv_role=kv_both`, two instances sharing one segment.
2. Send two different requests that share a long prefix but diverge **within the same 1152-token block** (e.g. same system prompt; different user task/question).
3. Observe the second request reusing external KV (`External prefix cache hit rate > 0`, `memory_keys>0`) and generating content that belongs to the first request.

## Workaround

Setting `--prefix-match-unit 1152` forces `hash_block_size == attention block size`, making `block_size > hash_block_size` false → `partial_hash_hits_enabled` returns false → only full, content-identical 1152-token blocks are reused, eliminating the contamination. It keeps prefix caching and cross-instance reuse (for fully identical prefixes), at the cost of a coarser (1152-token) matching granularity and a small loss of reuse on non-1152-multiple shared tails.

## Suggested fix (connector side)

In `partial_hash_hits_enabled`/`get_cached_block`/`align_lookup_length`, do **not** report a partial (sub-block) hit unless the full reused KV block is content-identical to the current request at that position. Concretely:
- Either treat the existence check at full-block (attention block) granularity regardless of `hash_block_size`, so a partial match cannot map onto a mismatched full block; or
- Gate partial hits so they only apply when `hash_block_size` is a divisor that coincides with the actual reusable KV block content boundaries, and verify content identity before reuse.

This likely affects MLA / multi-KV-group models (e.g. GLM5Next, DeepSeek V4-Flash) when `prefix-hash block (64) != attention block (1152)`. A secondary contributor worth noting alongside #2827 is the shared-store being near-full with SSD offload eviction mostly failing, which can also destabilize object existence and enable longer-than-valid reuse.

### Before submitting...

- [ ] Ensure you searched for relevant issues and read the [documentation]

Start with vllm/distributed/kv_transfer/kv_connector/v1/mooncake/store/coordinator.py and inspect partial_hash_hits_enabled, get_cached_block, and align_lookup_length. Reproduce with two requests that diverge within one 1152-token block while sharing a prefix, then trace whether a 64-token existence hit causes full-block reuse. Done means only content-identical full KV blocks are reused, with the reported contamination no longer reproducible.

Written by the indexing model from the issue text.