[Ticket]: In-place restart of the vLLM container always fails once — VRAM not yet released, error blames gpuMemoryUtilization

### Type

Bug

### Summary

When the kubelet restarts the **`model` container in place** — a liveness-probe failure or an engine crash, i.e. the platform's own self-healing path — the replacement vLLM process starts before the driver has reclaimed the dying process's VRAM. It fails immediately with what looks like a sizing error, back-offs, and only succeeds on the **third** attempt.

```
ValueError: Free memory on device cuda:0 (7.65/19.67 GiB) on startup is less than
desired GPU memory utilization (0.9, 17.71 GiB). Decrease GPU memory utilization or
reduce GPU memory used by other processes.
```

Net effect: **automatic recovery takes about twice as long as it should**, and the log blames GPU memory — which reads as "the model no longer fits" rather than "we restarted too fast".

### Intent

Self-healing should heal on the first attempt. Today the one path that exists to recover a wedged engine is also the path that is guaranteed to fail once, and the error it prints sends the reader toward `gpuMemoryUtilization` — a value that is correct and should not be touched.

### Source of truth (links)

- Measured evidence and full timeline: https://github.com/ADORSYS-GIS/inference-ops/blob/main/docs/benchmarks/2026-09-03-qwen3-5-2b-track0-cache-probe.md (§6)
- Context PR: https://github.com/ADORSYS-GIS/inference-ops/pull/29
- The rollout that proved the scope is narrower than first assumed: https://github.com/ADORSYS-GIS/ai-helm-values/pull/373

### Current Behavior

Observed on `qwen3-5-2b` / `hetzner-k8s-gpu-2`, 2026-09-03, triggered by `kill -TERM 1` in the `model` container (the same in-place restart the kubelet performs):

| Time | Event |
|---|---|
| 17:18:00 | container restarted, `restarts=1` |
| 17:18:46 | attempt 2 starts |
| 17:19:22 | **attempt 2 exits 1** after 36 s — `Free memory on device cuda:0 (7.65/19.67 GiB)` |
| — | `Warning BackOff — Back-off restarting failed container model` |
| 17:19:43 | attempt 3 starts |
| ~17:22 | Ready. `restarts=3`, **~4 min** against a ~2 min model load |

The `lmcache` sidecar was untouched throughout (`restarts=0`), and the pod was never recreated — same pod name, same IP.

**This is not a manual-only scenario.** The `model` container carries a livenessProbe:

```yaml
livenessProbe:
tcpSocket: {port: 8080}
periodSeconds: 30
failureThreshold: 6 # ⇒ 3 minutes of an unreachable port, then an in-place restart
```

So any engine hang reaches exactly this code path: 3 min to detect, then a recovery that burns a guaranteed failed attempt.

### Expected Behavior

An in-place restart of the `model` container succeeds on the **first** attempt. `restartCount` increases by exactly 1, and no `BackOff` event is emitted.

### Acceptance Criteria

- [ ] `kubectl exec -c model -- kill -TERM 1` brings the container back with `restartCount` incremented by **exactly 1** and no `BackOff` event.
- [ ] Time from SIGTERM to `Ready` is within ~30 s of a cold pod start, not ~2×.
- [ ] If VRAM genuinely never frees, the container fails with a message naming **the wait**, not `gpuMemoryUtilization` — the operator must not be pointed at a correct value.
- [ ] `helm template charts/inference` output is unchanged for models where the fix is not applicable (llama.cpp, LocalAI).
- [ ] The behaviour is covered in `docs/` so the next person does not re-diagnose it as a sizing problem.

### Out of Scope

- **ArgoCD rollouts. They are not affected and need no change.** `strategy: Recreate` terminates the pod completely, so the GPU is released before the replacement schedules — verified on the 2026-09-03 L1 8→32 GiB rollout: `restarts=0`, ~3.5 min, clean. Any fix must not slow this path down.
- `gpuMemoryUtilization` itself. 0.9 is correct; the KV pool depends on it and lowering it to dodge the race would cost real capacity.
- The LMCache sidecar's own GPU footprint (`NVIDIA_VISIBLE_DEVICES=all`, ~1 GB `kv_buffer_size`). It is a small constant, present in both the working and failing cases, and not the cause.
- Liveness-probe tuning. Raising `failureThreshold` delays detection; it does not fix the race.

### Technical Context

The kubelet restarts a container **inside the existing pod sandbox**. Device-plugin allocations belong to the *pod*, so the GPU stays assigned and the new process starts within seconds of the old one's exit — well before the driver has reclaimed ~12 GiB. vLLM checks free VRAM once, at startup, and exits rather than waiting.

**The fix cannot be an `initContainer`.** Ordinary init containers run once per pod sandbox and do **not** re-run when a single container restarts — which is the only case that fails here. It has to live in the `model` container's own startup path.

Suggested shape — a pre-flight wait in the entrypoint, wrapping the existing args:

```yaml
command: ["/bin/sh", "-ec"]
args:
- |
need_mib=$(( $(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits) * 90 / 100 ))
for _ in $(seq 1 60); do
free_mib=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits)
[ "$free_mib" -ge "$need_mib" ] && break
echo "waiting for VRAM: ${free_mib}MiB free, need ${need_mib}MiB (a prior process is still releasing)"
sleep 2
done
exec vllm serve "$@"
- -- # then the existing inference.serverArgs
```

Notes for whoever picks this up:

- `nvidia-smi` is present in the model image — verified `/usr/bin/nvidia-smi` in `lmcache/vllm-openai:nightly-2026-08-18-cu129`.
- The `90` must be derived from `serving.gpuMemoryUtilization`, not hardcoded.
- The timeout must fail **loudly and specifically**; a silent fall-through would reproduce today's confusing error with extra latency in front of it.
- On a cold pod start the loop exits on its first iteration, so the ArgoCD path is unaffected.
- `charts/inference/templates/_helpers.tpl` currently passes the engine command line via `args:` and leaves the image entrypoint in place (`inference.serverArgs`, ~L155/L770). Introducing `command:` there needs care — the same file already warns that an empty `args:` renders as YAML null and blanks the image's command.

### Risks

- **Touching the entrypoint of every vLLM model is riskier than the bug it fixes** if it is done bluntly. Scope it to the vLLM engine path and leave llama.cpp/LocalAI alone.
- A wrong threshold turns a 1-in-N slow restart into a model that never starts. Fail-loud on timeout, and keep the timeout well above the observed ~40 s reclaim.
- `nvidia-smi` availability is an image property; it is verified for the current tag only and would need re-checking on an image bump.

### Test Plan

1. `kubectl exec -c model -- kill -TERM 1`; assert `restartCount` +1, no `BackOff` event, and the wait line appears in the log at most a few times.
2. Repeat 3× to rule out timing luck.
3. Trigger the liveness path directly (block :8080 inside the container) and confirm the same clean single restart.
4. Delete the pod (ArgoCD-equivalent path) and confirm no regression: the wait loop exits immediately and start-up time is unchanged.
5. `helm template charts/inference` diffed against `main` — only the vLLM model containers change.

### Verification evidence

Reproduce the failure as it stands today:

```bash
POD=$(kubectl --context hetzner-prod -n inference get pods \
-l app.kubernetes.io/name=inference-qwen3-5-2b -o jsonpath='{.items[0].metadata.name}')
kubectl --context hetzner-prod -n inference exec "$POD" -c model -- kill -TERM 1
kubectl --context hetzner-prod -n inference get pod "$POD" \
-o jsonpath='{.status.containerStatuses[?(@.name=="model")].restartCount}{"\n"}'
kubectl --context hetzner-prod -n inference logs "$POD" -c model --previous | grep -i "Free memory on device"
kubectl --context hetzner-prod -n inference get events --field-selector involvedObject.name="$POD" | grep BackOff
```

⚠️ That reproduction is a **~4 minute production outage** of the only self-hosted model the gateway serves (`strategy: Recreate`, one GPU, no HA). Do it deliberately or on a lab node.

_{Filed from measurements taken during the inference-ops Track 0 cache probe. The defect was found as a side effect of testing something else, and its scope was narrowed afterwards when an ArgoCD rollout did **not** reproduce it — the original write-up claimed "every restart", which was too broad.}