MCPServer with backendReplicas > 1: hardcoded ClientIP affinity on the backend Service starves all but one backend pod

Summary

When an MCPServer runs with backendReplicas > 1, the proxy runner creates the backend ClusterIP Service (mcp-<name>) with a hardcoded sessionAffinity: ClientIP (1800s) and uses it as its target. Because the proxy pods are the Service's only clients, and the proxy's health checks (GET / every few seconds) keep the kube-proxy affinity entry perpetually fresh, each proxy pod pins all of its traffic — including every new session's initialize — to a single backend pod, indefinitely.

Observed in production (operator + proxyrunner v0.32.0, EKS, replicas: 2, backendReplicas: 2, Redis session storage): one backend StatefulSet pod served 100% of requests while the other received zero requests for 4+ days (not even health checks). Both proxy pods happened to pin to the same backend. We saw the same on a second MCPServer in the cluster. The idle replica is pure cost: it only helps as a cold spare on failover.

Why the affinity no longer pays its way

The hardcoded affinity predates the per-session backend routing work. Since #4574 / the session-storage work, per-session stickiness is handled at the proxy layer:

• initialize is forwarded to the ClusterIP; the actual pod is captured via httptrace.GotConn and stored as backend_url in (Redis-backed) session metadata (pkg/transport/proxy/transparent/transparent_proxy.go).
• Follow-up requests route directly to that pod, surviving proxy restarts, with transparent re-initialization if the pod is replaced.

So L7 session stickiness is already guaranteed without any Service-level affinity. The L4 ClientIP affinity adds no correctness — it only collapses the initialize distribution to at most one backend per proxy pod (and in practice fewer, since pins are random and never expire under continuous health checking). The code comment acknowledges this: "this provides proxy-runner-level stickiness (L4), not per-MCP-session stickiness (L7)" (pkg/container/kubernetes/client.go, mcpServiceSessionAffinityTimeout).

Note this is a different Service from the one discussed in #4122 — that's client→proxy affinity (spec.sessionAffinity, which is configurable on the CRD). The backend mcp-<name> Service affinity has no knob: createMCPService passes sessionAffinity: true unconditionally (still the case on main), and MCPServer.spec.sessionAffinity only applies to the proxy Service.

Reproduction

1. Deploy an MCPServer (streamable-http) with replicas: 2, backendReplicas: 2, Redis sessionStorage.
2. Drive sessions through the proxy from any number of clients.
3. Observe backend pod logs: all requests (health checks + MCP traffic) land on the pod(s) each proxy first connected to; the affinity entry never expires because health checks continuously refresh it. With luck, one backend pod receives nothing at all.

Ask

Make the backend Service affinity configurable — any of these would work:

• Skip/disable the ClientIP affinity on mcp-<name> when backendReplicas > 1 and session storage is configured (the case where backend_url pinning already provides stickiness) — arguably the right default;
• or add a CRD field (e.g. spec.backendSessionAffinity) mirroring spec.sessionAffinity;
• or honor spec.sessionAffinity for both Services.

Workaround

We currently run a CronJob that patches spec.sessionAffinity: None onto the backend Services every 10 minutes — a plain kubectl patch doesn't stick because the proxyrunner re-asserts ClientIP via server-side apply (Force) each time a proxy pod starts. After the patch, new sessions distribute across both backend pods and existing sessions are unaffected (still routed via backend_url).

Environment

• toolhive operator + proxyrunner v0.32.0 (ghcr.io/stacklok/toolhive/*:v0.32.0)
• EKS (us-gov-west-1), kube-proxy iptables mode
• MCPServer: streamable-http, replicas: 2, backendReplicas: 2, Redis sessionStorage, embedded auth server

Related: #4575 / #4484 (backend_url session routing), #4122 (proxy-side affinity discussion)

Happy to send a PR if maintainers agree on the preferred shape (default-off when session storage is present vs. new CRD field).