[Usage]: Incomplete SSD-backed KV reuse after eviction with MooncakeStoreConnector

### Describe your usage question

### Describe your usage question

With a 64 GB Mooncake memory pool and SSD offload enabled with 2 TB pool, I stored three distinct
50,000-token prompts sequentially. I then cleared only vLLM's local GPU prefix cache
and replayed the same prompts.

I expected nearly all 50,000 tokens of each prompt to be reused from Mooncake memory
or SSD. Instead, external reuse was 49,984, 16,384, and 0 tokens respectively.

### Environment

- GPU: 2 x NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB each)
- NVIDIA driver: 580.173.02
- Base image: `vllm/vllm-openai:v0.26.0`
- vLLM: `0.26.0`
- `mooncake-transfer-engine`: `0.3.12.post1` from PyPI
- Transport: TCP
- Model: Gemma 4 31B, TP=2
- SSD: local NVMe array mounted at `/mnt/nvme0`; an independent sequential-write
test sustains approximately 4.8 GB/s

The package was installed directly into the official vLLM image:

```dockerfile
FROM vllm/vllm-openai:v0.26.0
RUN python3 -m pip install --no-cache-dir \
mooncake-transfer-engine==0.3.12.post1 \
nvidia-cuda-runtime-cu12
```

### Minimal setup

Master:

```bash
mooncake_master \
--port 50051 \
--enable_offload=true \
--offload_on_evict=true \
--default_kv_lease_ttl=60s
```

Standalone store owner:

```bash
MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/mnt/nvme0/mooncake-issue-repro \
MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=68719476736 \
mooncake_client \
--host 127.0.0.1 \
--port 50053 \
--master_server_address 127.0.0.1:50051 \
--metadata_server P2PHANDSHAKE \
--protocol tcp \
--global_segment_size 64GB \
--local_buffer_size 4GB \
--enable_offload=true
```

vLLM:

```bash
MOONCAKE_CONFIG_PATH=/mooncake_config.json \
MOONCAKE_PREFERRED_SEGMENT=127.0.0.1:50053 \
VLLM_MOONCAKE_STORE_TIER_LOG=1 \
VLLM_SERVER_DEV_MODE=1 \
VLLM_USE_V2_MODEL_RUNNER=0 \
PYTHONHASHSEED=0 \
vllm serve /model \
--served-model-name gemma-4-31b-it \
--tensor-parallel-size 2 \
--max-model-len 65536 \
--enable-prefix-caching \
--kv-transfer-config \
'{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"enable_cross_layers_blocks":true}}'
```

### Reproduction

1. Start with an empty SSD directory.
2. Send three distinct prompts sequentially, each containing exactly 50,000 token IDs.
3. Wait 60 seconds after each request and another 120 seconds after all three.
4. Clear only the local GPU prefix cache:

```bash
curl -X POST http://127.0.0.1:8000/reset_prefix_cache
```

5. Replay the same three prompts in the same order.

All six inference requests completed successfully.

### Actual result

Before replay, Mooncake reported:

```text
master_allocated_bytes 61766369280
master_allocated_file_size_bytes 56953405440
master_key_count 52428
master_batch_put_end_items_total 52428
mem_cache_nums_ 47124
file_cache_nums_ 43452
master_evicted_key_count 5304
```

Raw SSD I/O is unlikely to be the bottleneck. Mooncake persisted about 56.95 GB,
and the key counts remained unchanged during an additional 120-second settle period.
At the measured local SSD throughput, this amount of data should complete well within
that window.

After resetting the GPU cache, local cache hits remained zero. External hit-token
deltas for the three replayed prompts were:

- Prompt 1: `49,984 / 50,000` (99.97%)
- Prompt 2: `16,384 / 50,000` (32.77%)
- Prompt 3: `0 / 50,000` (0%)
- Total: `66,368 / 150,000` (44.25%)

The raw vLLM metrics before replay were:

```text
vllm:prefix_cache_hits_total{engine="0",model_name="gemma-4-31b-it"} 0.0
vllm:external_prefix_cache_queries_total{engine="0",model_name="gemma-4-31b-it"} 150000.0
vllm:external_prefix_cache_hits_total{engine="0",model_name="gemma-4-31b-it"} 0.0
```

After all three replay requests:

```text
vllm:prefix_cache_hits_total{engine="0",model_name="gemma-4-31b-it"} 0.0
vllm:external_prefix_cache_queries_total{engine="0",model_name="gemma-4-31b-it"} 300000.0
vllm:external_prefix_cache_hits_total{engine="0",model_name="gemma-4-31b-it"} 66368.0
vllm:prompt_tokens_by_source_total{engine="0",model_name="gemma-4-31b-it",source="local_compute"} 233632.0
vllm:prompt_tokens_by_source_total{engine="0",model_name="gemma-4-31b-it",source="local_cache_hit"} 0.0
vllm:prompt_tokens_by_source_total{engine="0",model_name="gemma-4-31b-it",source="external_kv_transfer"} 66368.0
```

### Relevant package logs

The first prompt was loaded successfully from both memory and disk:

```text
(Worker_TP0 pid=468) INFO 08-07 07:41:11 [worker.py:331] Mooncake load tier summary: req_id=cmpl-97b9984421b9d2ef-0-94181581 batch_keys=915 memory_keys=849 disk_keys=66 unknown_keys=0 success_keys=915 failed_keys=0 bytes_by_tier={'memory': 1112801280, 'disk': 86507520, 'unknown': 0}
(Worker_TP1 pid=469) INFO 08-07 07:41:12 [worker.py:331] Mooncake load tier summary: req_id=cmpl-97b9984421b9d2ef-0-94181581 batch_keys=915 memory_keys=793 disk_keys=122 unknown_keys=0 success_keys=915 failed_keys=0 bytes_by_tier={'memory': 1039400960, 'disk': 159907840, 'unknown': 0}
```

During the initial store phase, vLLM/Mooncake reported allocation failures:

```text
W0807 07:37:10.594792 1171 client_service.cpp:2461] BatchPut failed for 2816 keys due to insufficient space. Consider lowering eviction_high_watermark_ratio or mounting more segments.
W0807 07:37:12.530093 1170 client_service.cpp:2461] BatchPut failed for 1648 keys due to insufficient space. Consider lowering eviction_high_watermark_ratio or mounting more segments.
W0807 07:37:47.717445 1170 client_service.cpp:2461] BatchPut failed for 2816 keys due to insufficient space. Consider lowering eviction_high_watermark_ratio or mounting more segments.
W0807 07:37:47.717864 1171 client_service.cpp:2461] BatchPut failed for 2816 keys due to insufficient space. Consider lowering eviction_high_watermark_ratio or mounting more segments.
```

The standalone client also logged:

```text
E0807 07:37:56.937430 156 file_storage.cpp:624] Failed to store objects with error: OBJECT_ALREADY_EXISTS
E0807 07:37:56.945380 156 file_storage.cpp:844] Failed to persist objects with error: OBJECT_ALREADY_EXISTS
```

Is this expected backpressure behavior with `offload_on_evict=true`? If not, is there
a required setting that makes `BatchPut` wait/retry until eviction and SSD persistence
free enough memory, instead of leaving later prompts only partially reusable?

### Before submitting a new issue...

- [x] I searched existing issues and read the documentation.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)