[RFC]: Control-plane orchestration of vLLM sleep mode and kvcached for warm standby and multi-model GPU pooling

### Summary

AIBrix scales replicas as 0-or-N and only *observes* engine memory state; it never drives it. Two engine-side mechanisms now make GPU memory reclaimable at runtime — vLLM **sleep mode** (park a whole idle model's weights+KV) and **kvcached** (elastically shrink/grow a model's KV so several models share one GPU) — but both are mechanism-only, with no control plane. This RFC proposes AIBrix own the *policy*: a control-plane-driven "warm/asleep" tier between one replica and zero, plus KV-budget arbitration across co-resident models. The result: multi-minute cold starts become second-scale wakes, and one GPU can densely host many long-tail models.

### Motivation

Scaling is binary today. To reclaim a GPU we either keep a replica running (burning HBM while idle) or delete the pod and pay a multi-minute cold start (weight load + CUDA graph capture) when traffic returns. Neither fits bursty or long-tail traffic.

The engine now exposes the levers, but not the policy:

| | vLLM sleep mode | kvcached (github.com/ovg-project/kvcached) |
|---|---|---|
| Scope | whole model (weights + KV) | KV cache only (weights allocated outside it) |
| Granularity | all-or-nothing, single process | page-level elastic, cross-process co-residency |
| Target | a fully **idle** model | several **active** models sharing a GPU |
| Mechanism | CUDA VMM; `/sleep` `/wake_up`; level 1 weights→CPU, level 2 discard | CUDA VMM; per-process `cudaMemGetInfo` headroom + `/dev/shm` budget via `kvctl` |
| Control | manual API only; `VLLM_SERVER_DEV_MODE`-gated; no autoscaling | decentralized by design; no scheduler/admission/cross-node; `controller/` is a demo |

They are complementary: kvcached handles elastic KV among active co-resident models; sleep parks idle models' weights. Both are VMM actuators that enforce/observe — neither *decides*. The missing piece is a control plane that decides **which** model sleeps/wakes and **when**, **how much** KV budget each co-resident model gets, **admission** when a GPU is oversubscribed, and **routing/readiness** as models move between awake / warm / asleep. AIBrix already scrapes `vllm:engine_sleep_state` but never acts on it, and has no notion of co-resident KV budget — so it is the natural owner.

### Proposed Change

(Sketch only; full design — CRD, state machine, budget algorithm, agent protocol — to follow in a complete doc.)

1. A **warm/asleep lifecycle state** between `minReplicas` and zero: low traffic → `POST /sleep?level=1&mode=wait` (graceful drain); traffic returns → `/wake_up` (seconds). Pod/IP/endpoint persist.
2. A **KV-budget arbiter** across co-resident models, driving `kvctl` limits by policy (per-model min/max/share, priority) with cost-asymmetric reclaim — shrink KV first (cheap), sleep weights only when protected floors don't fit (expensive).
3. A **decide/execute split**: a Go controller (placement, admission, sleep/wake & budget policy, status) over a pod-side agent (kvcached engine, `kvctl`, sleep).
4. **Required couplings:** invalidate the prefix index on sleep (KV is wiped — see the sleep/prefix-index bug), exclude asleep pods from routing/readiness, and document the `VLLM_SERVER_DEV_MODE` requirement.

### Alternatives Considered

- **HPA scale-to-zero** (`minReplicas=0`): already supported, but pays the full cold start and has no warm tier.
- **Leave it to kvcached's decentralized scheme:** works on a single node but has no admission/priority/cross-node policy and no sleep orchestration — precisely the demo `controller/` AIBrix would replace.
- **Token-level time-sharing** (swap at request granularity): higher utilization ceiling but far more complex/intrusive; this RFC targets model-granularity pooling.