Handle multi-pod refresh token rotation race condition

Context

PR #4117 extracted upstream token refresh into a reusable InProcessService (pkg/auth/upstreamtoken). During review, a race condition was identified that affects multi-pod deployments with shared Redis storage and single-use refresh token rotation at the upstream IDP.

The singleflight.Group in InProcessService deduplicates concurrent refresh attempts within a single pod, but cannot coordinate across pods. When two pods simultaneously detect an expired access token and attempt to refresh using the same single-use refresh token, only one succeeds — the other gets rejected by the IDP.

The Race

Time    Pod A                              Pod B                              Redis
─────   ──────────────────────────────     ──────────────────────────────     ─────────────
t0      GetValidTokens(sid)                                                  {AT_old, RT1}
t1      storage.Get → expired + RT1        GetValidTokens(sid)               {AT_old, RT1}
t2                                         storage.Get → expired + RT1       {AT_old, RT1}
t3      provider.RefreshTokens(RT1)        provider.RefreshTokens(RT1)       {AT_old, RT1}
t4      IDP returns AT2 + RT2                                                {AT_old, RT1}
t5      storage.Store(AT2, RT2)            IDP rejects RT1 → error           {AT2, RT2}
t6                                         → ErrRefreshFailed → 401

The race window is approximately 50–320ms (one Redis GET + one IDP round-trip + one Redis SET).

Current Mitigation

SessionAffinity (shipped in #3992) is the primary defense. With SessionAffinity: ClientIP, kube-proxy pins a client IP to the same backend pod, so the in-process singleflight handles deduplication. This eliminates the race in the common case.

SessionAffinity breaks during:

• Rolling deployments / pod evictions
• Client IP changes (mobile network switch, VPN reconnect)
• L7 load balancers that mask client IPs (Ingress, service mesh)

Risk Assessment

Current risk is low:

• MCPServer runs single-replica today with an embedded auth server
• vMCP does not use upstream token refresh yet
• SessionAffinity covers the common case

Risk increases when:

• vMCP adds upstream token injection (multi-replica is the default)
• Users deploy behind L7 load balancers without client-IP-based affinity
• IDPs with aggressive replay detection are used (see below)

IDP-Specific Behavior

Not all IDPs handle refresh token replay the same way:

IDP	Behavior on replay of consumed RT	Severity
Okta	Revokes entire grant family (all ATs + RTs for that session)	Critical — both pods lose tokens
Auth0	Configurable "Reuse Interval" (default: no grace)	High if interval is 0
Keycloak	Configurable; no grace period when rotation enabled	High when enabled
Entra ID	~10s grace period for confidential clients	Low
Google	Doesn't rotate RTs by default	N/A

Since ToolHive is IDP-agnostic, the design must assume worst-case (Okta).

Proposed Mitigation Tiers

Tier 1: Retry-with-Reread (low effort, partial fix)

After RefreshAndStore fails, re-read Redis. If another pod already stored fresh tokens, use them instead of returning 401.

• ~10 lines in InProcessService.refreshOrFail
• No interface changes, no new dependencies
• Handles the case where the IDP doesn't punish replay
• Does not prevent the duplicate IDP call (still unsafe with Okta)

Tier 2: Redis distributed lock (moderate effort, complete fix)

Acquire a SET NX EX lock on {prefix}refresh-lock:{sessionID} before calling the IDP. Only the lock holder refreshes; other pods wait and re-read.

• Prevents duplicate IDP calls entirely (safe with Okta)
• Uses existing Redis infrastructure and Lua script patterns
• Self-contained in pkg/auth/upstreamtoken/service.go
• Failure mode: lock-holder crash → other pods wait until TTL expires (use 30s to match refreshTimeout)
• Lock acquisition failure (Redis down) → fail closed (consistent with existing behavior in 27644a8e)

Tier 3: CAS on storage (optional hardening)

Add CompareAndStoreUpstreamTokens to UpstreamTokenStorage — write only succeeds if the stored RT matches what was read. Prevents stale overwrites even with non-rotating IDPs.

• Extends existing Lua script in storeUpstreamTokensScript
• New method on UpstreamTokenStorage interface
• Complementary to Tier 2, not a replacement

Not recommended

• K8s Lease-based coordination: Wrong granularity (per-session coordination needs sub-second; Leases are per-workload with 10–15s renewal)
• Singleton refresh service: Operationally heavy, architectural mismatch with embedded auth server design
• Leader election: Concentrates all refresh work on one pod, creates single point of failure

Suggested Implementation Order

1. Tier 1 (retry-with-reread) as a quick safety net
2. Tier 2 (Redis lock) when vMCP adds upstream token support
3. Tier 3 (CAS) as optional hardening if needed

Relevant Files

• pkg/auth/upstreamtoken/service.go — InProcessService.refreshOrFail (primary change site)
• pkg/auth/upstreamswap/middleware.go — error-to-HTTP mapping
• pkg/authserver/refresher.go — RefreshAndStore (blind write, no CAS)
• pkg/authserver/storage/redis.go — storeUpstreamTokensScript Lua script
• pkg/authserver/storage/types.go — UpstreamTokenStorage interface (no CAS method)
• cmd/thv-operator/controllers/virtualmcpserver_deployment.go — SessionAffinity config

Additional Note

The operator should set explicit SessionAffinityConfig.ClientIP.TimeoutSeconds: 1800 on VirtualMCPServer services for parity with the non-operator path (pkg/container/kubernetes/client.go:1028).

Start with pkg/auth/upstreamtoken/service.go and trace refreshOrFail into pkg/authserver/refresher.go and the Redis storage implementation. Review the middleware error mapping and the existing Lua script patterns before deciding which mitigation tier is in scope. Done should include an agreed coordination approach, coverage for concurrent multi-pod refreshes, and preserved failure behavior.