[Ticket]: Spike — is an L2 disk tier the only way to a KV cache larger than APC? (gate on #4701 first)

### Type

Spike / Investigation

### Summary

An L2 (node-local disk) tier is the **only remaining way** to give this fleet a KV cache larger than vLLM's own in-GPU prefix cache. Measured 2026-09-03: LMCache's host-RAM L1 contributes **exactly zero** in steady state — not "little", zero tokens — because at 32 GiB it holds ~554,000 tokens against APC's 995,932 and is therefore a strict LRU subset of a cache that is checked first. Host RAM cannot fix this (parity needs ~58 GiB on a 62.6 GiB node that also runs Longhorn). The GPU nodes each have **~1.7 TB of ephemeral NVMe**, which could exceed `C_hbm` by orders of magnitude.

This spike answers: **is an L2 disk tier worth enabling, and is it safe on this model?**

### Intent

We have spent two rounds sizing a cache that structurally cannot help in steady state, and proven the ceiling. Rather than keep tuning L1, establish once whether the disk tier changes the regime — or close the question and accept that LMCache here is restart insurance only.

### Source of truth (links)

- Measurement that closes the L1 line: https://github.com/ADORSYS-GIS/inference-ops/blob/main/docs/benchmarks/2026-09-03-qwen3-5-2b-rfc0002-cache-ab.md
- The L1 sizing change and its rationale: https://github.com/ADORSYS-GIS/ai-helm-values/pull/373
- Correctness probe that cleared L1 only: https://github.com/ADORSYS-GIS/inference-ops/blob/main/docs/benchmarks/2026-09-03-qwen3-5-2b-track0-cache-probe.md
- Upstream bug that gates this: https://github.com/LMCache/LMCache/issues/4701

### Current Behavior

`--l1-size-gb 32`, no L2 tier configured. Measured contribution:

| Regime | APC | LMCache |
|---|---|---|
| Steady state, working set < HBM | 92.5 % hit, 7.68× | **0** |
| Steady state, working set = 1.33 × HBM | 21.5 % hit | **0** |
| First requests after a restart | 0 | 92.5 % hit, **6.57×** |

### Expected Behavior

A decision, backed by a number, on whether to enable an L2 disk tier — either measurable steady-state hits at a working set above `C_hbm`, or a written "no" that closes the question.

### Acceptance Criteria

- [ ] **⚠️ Correctness gate FIRST.** Re-run the Track 0 buried-fact probe **with the disk tier enabled**, in a lab or on a card that is not serving users. LMCache #4701's primary reproduction *used an L2 disk tier*; Track 0 cleared this fleet **for L1 only** and explicitly says adding a disk tier re-opens the question. **Do not proceed to performance work until this passes.**
- [ ] Steady-state re-run of the RFC-0002 Phase 3 sweep (working set ≥ 1.33 × `C_hbm`) with L2 enabled; report `external_prefix_cache_hits_total` and TTFT by hit source.
- [ ] Compare a disk-tier restore against the measured host-RAM restore (0.088 s median for a 5,321-token prefix) and against a full prefill (0.578 s). A disk restore slower than a prefill is a no.
- [ ] Page-cache confound controlled — a second read served from Linux page cache is not a disk-tier measurement. Either drop caches between runs or size the working set beyond RAM, and say which.
- [ ] SSD wear and capacity impact assessed against the ~1.7 TB ephemeral volume; a retention/eviction policy proposed.
- [ ] Outcome written up as a dated benchmark report and an ADR, whichever way it lands.

### Out of Scope

- Raising L1 further. 58 GiB is the parity threshold and the node cannot safely provide it; this is settled, not deferred.
- Any remote or shared cache tier. Ruled out on bandwidth grounds — the boxes have a 1 GBit/s port and a vSwitch shares it. See inference-ops ADR-0010.
- The restart-recovery behaviour, which already works and is worth keeping.

### Technical Context

- `C_hbm` = **995,932 tokens** (11.55 GiB), read from the vLLM startup log — not derived. Any tier meant to help steady state must exceed it.
- L1 costs **~62,000 B/token** empirically, ~5× APC's 12,446 B/token, because LMCache stores all 24 pages per chunk including the 18 GDN recurrent-state snapshots while HBM pages only the 6 full-attention layers.
- At that rate, 1.7 TB of NVMe ≈ **27 M tokens**, ~27× `C_hbm`. Capacity is not the constraint; correctness and restore latency are.
- The node's ephemeral storage is **not** Longhorn and must not become a Longhorn volume — the fleet is already below Longhorn's documented 3-node / 10 Gbps spec.
- Relevant existing note: inference-ops `reference/kv-cache-offloading-part2-technical.md` §3.2 already flags SSD wear, and §3.3 cold-start latency.

### Risks

- **Silent wrong answers.** This is the configuration LMCache #4701 describes. A disk tier that reports high hit rates and returns corrupted KV is worse than no cache at all, and the failure is invisible without a buried-fact probe.
- Measuring the Linux page cache and calling it a disk tier — the classic false positive here.
- SSD wear on a node whose ephemeral volume also holds container images and model weights.
- Opportunity cost: if the answer is "restore is slower than prefill", the spike returns a no. That is a valid outcome and should be closed quickly rather than tuned.

### Test Plan

1. Lab/non-serving card. Enable `--l2-*` disk tier alongside L1.
2. Buried-fact probe, cold vs restored, with a control set — same method as Track 0 §5. **Gate.**
3. Phase 3 sweep at ≥ 1.33 × `C_hbm`; record hits by source and TTFT.
4. Restart arm with the disk tier, to check it does not regress the one regime that currently works.
5. Report; ADR either way.

### Verification evidence

Baseline to beat, measured 2026-09-03 (all in-pod, concurrency 1, 5,321-token prefixes):

```text
full prefill, no cache median 0.578 s
LMCache L1 restore median 0.088 s (92.5 % hit, APC empty)
APC in-GPU hit median 0.073 s
steady-state LMCache hits 0 tokens at W = 1.33 x C_hbm
L1 capacity ~554,000 tokens vs C_hbm 995,932 (56 %)
```