Phase 2: Web Distributed Locking for Multi-Replica Scaling

Problem

The web Passenger connection pool has a connection-to-process ratio of 85:1 (13125 MAX_REQUESTS : 154 MAX_POOL). This means HTTP connections queue rapidly when 10k workers complete jobs and POST results back simultaneously. Scaling to 3+ web replicas requires distributed file locking (B4).

What Phase 1 Did

• Added web_hpa.distributedLocking config section in values.yaml and values.schema.json
• Scaled web HPA minReplicas: 1→2 and maxReplicas: 1→6
• Updated web passenger_memory_per_process documentation

What Remains (This Issue)

1. Application-level Redis redlock: Implement redlock-rbs or equivalent in the OpenStudio Server Rails app so file-level locks work across multiple web pods
2. NFS noac mount option: Ensure NFS mount on web pods uses noac (no attribute caching) for cross-pod consistency
3. Validation: Write stress test that sends 10k concurrent POST requests to /analyses with distributed locking enabled, verifying no race conditions
4. Documentation: Update QUICKSTART and cloud-specific READMEs with multi-web-replica guidance
5. Migration path: Phased rollout — single replica (default) → 2 replicas with locking → N replicas

Design Notes

• Lock key pattern: redlock:<analysis_id>:<step> in Redis (same Redis instance used for Resque)
• Lock TTL should be short (5-30s) with auto-renewal for long operations
• The distributedLocking config in values.yaml is a signal only — the actual locking must be implemented in the app
• Web HPA can scale beyond 1 replica only after locking is validated

References

• .agents/results/result-tf-infra.md — B4: Web Passenger Connection Pool (CRITICAL)
• openstudio-server/values.yaml — web_hpa.distributedLocking config
• openstudio-server/templates/web/web-hpa.yaml
• redlock-rbs

Acceptance Criteria

• Redis redlock implemented in OpenStudio Server application
• Lock acquisition + release verified under concurrent POST load
• NFS mount includes noac option for multi-pod consistency
• Web HPA can scale to 3+ replicas without race conditions
• Document procedure for rolling out multi-web-replica

Start with openstudio-server/values.yaml, values.schema.json, and templates/web/web-hpa.yaml, then inspect the Rails application's existing Resque Redis integration and file-locking paths. Review the B4 result and the redlock-rbs reference before defining the lock behavior. Done means concurrent POST validation, NFS noac configuration, multi-replica rollout guidance in QUICKSTART and cloud READMEs, and a safe single-replica-to-N-replica migration path.