Phase 2: Redis Sentinel HA Deployment for 10k-worker scaling

Problem

The single Redis pod (RWO PVC, Recreate strategy) is the hardest scaling ceiling at ~2k-3k workers. Each worker maintains a persistent BRPOP connection; at 10,000 concurrent connections, Redis CPU saturates before workers are fully utilized. Single-pod Redis is also a SPOF — any restart kills all queue operations.

What Phase 1 Did

• Scaled Redis CPU 3→8 cores, memory 4Gi→16Gi
• Set maxclients: 50000 and tcpBacklog: 511
• Tuned appendfsync: everysec and disabled RDB saves (save: "")
• Scaffolded ha.mode: standalone|sentinel in values.yaml with sentinel sidecar container in redis-deploy.yaml
• Added redis.ha.sentinel config block with quorum/failover/parallel-syncs

What Remains (This Issue)

1. Sentinel deployment template: Convert the current single-pod Deployment into a 3-pod StatefulSet with headless service for stable network identities (pod-N.service)
2. Sentinel discovery service: Create a redis-sentinel service (ClusterIP or headless) so workers discover the current Redis primary
3. Primary/replica discovery: Add an init container or sidecar that queries sentinel to resolve the current primary and populates REDIS_URL dynamically
4. Read splitting (optional): Route BRPOP to primary, read-only queries to replicas
5. Validation: Verify automatic failover works when the primary pod is deleted

Design Notes

• Sentinel quorum of 2 (with 3 replicas) prevents split-brain
• Worker pods must be updated to query sentinel for primary address on startup
• The sentinel sidecar is already scaffolded in redis-deploy.yaml but the full StatefulSet + headless service template is needed
• RWO PVC per pod (not shared) — each has its own AOF/RDB volume
• Phase 1 HA mode is wired but disabled by default (mode: standalone) to maintain backward compat

References

• .agents/results/result-tf-infra.md — B1: Single Redis Instance (CRITICAL)
• openstudio-server/values.yaml — redis.ha config block
• openstudio-server/templates/redis/redis-deploy.yaml — sentinel sidecar scaffold
• openstudio-server/values.schema.json — HA mode validation

Acceptance Criteria

• 3-pod Redis StatefulSet with stable network identities
• Sentinel sidecar auto-configures primary/replica
• Workers discover primary via sentinel, not hardcoded DNS
• Automatic failover completes in < 30s when primary is killed
• No data loss during failover (AOF appendfsync everysec)
• Backward compatible — standalone mode still works when ha.mode: standalone

Start by reviewing openstudio-server/values.yaml, values.schema.json, and templates/redis/redis-deploy.yaml, then trace how workers receive REDIS_URL. Implement and validate the StatefulSet, services, Sentinel-based discovery, and standalone compatibility against the listed acceptance criteria, including failover and data-loss checks.