container-cache: HTTP liveness probe converts heavy load into restart storms; use TCP liveness

Component

deploy/helm/container-cache (nginx proxy cache StatefulSet)

Problem

The nginx-proxy container uses an HTTP liveness probe against /healthz. That endpoint is served by the same nginx worker processes that serve the data plane, so under sustained heavy transfer load (many concurrent large model downloads filling the cache) the probe times out even though the proxy is healthy and actively serving. kubelet then kills the container.

Each liveness kill makes things worse, not better:

1. Every in-flight transfer through that replica is aborted; all downloading clients re-request, re-saturating the tier (retry herd).
2. Clients whose connections broke re-establish their pools against the surviving replicas, so traffic stays skewed onto the survivors long after the killed pods return Ready.
3. The surviving replica also retains the full backlog of in-flight/lock-waiting requests from the dead clients and drains it slowly through upstream/lock timeout ladders, while restarted replicas start clean - an invisible imbalance lasting tens of minutes.

Load testing reproduced the kill deterministically at herd scale (tens of workers downloading a multi-hundred-GB model concurrently) with node network and backing storage independently verified healthy - the probe configuration alone converts "slow" into "restart storm". A previous mitigation that only loosened the HTTP liveness thresholds (60s to kill) was not sufficient.

Proposed fix

• Liveness: replace the HTTP probe with a tcpSocket probe on the same port. A TCP accept is completed by the kernel while nginx holds the listener, so liveness fails only when the process is actually dead or wedged - a busy proxy is never killed.
• Readiness: keep the HTTP /healthz probe (slightly more tolerant timeout/threshold) so an overloaded replica sheds new traffic without being killed. Degradation stays possible; self-inflicted crashes do not.
• Keep both configurable via the existing probes: values block.

Acceptance

Herd-scale download test against the cache tier completes (or degrades in latency only) with zero cache pod restarts and no probe-kill events.

Start in deploy/helm/container-cache and inspect the nginx proxy cache StatefulSet and existing probes: values block. Replace only the liveness behavior as proposed while retaining the HTTP readiness probe and its configurability. Done means a herd-scale download completes without cache pod restarts or probe-kill events.