Phase 2: MongoDB Replica Set StatefulSet for 10k-worker scaling

Problem

The single MongoDB pod (RWO PVC, Recreate strategy) is a critical SPOF with ceiling at ~3k-5k workers. Each worker writes 3+ status updates per job (queued→started→completed), generating ~30,000 writes/minute at 10k workers. A single pod faces write lock contention under 10k concurrent DataPoint updates, and any restart stalls all job progress tracking.

What Phase 1 Did

• Nothing yet — MongoDB was not modified in Phase 1

What Should Be Done (This Issue)

1. Replica Set StatefulSet: Create a 3-pod StatefulSet template (primary + secondary + arbiter) with headless service
2. Connection string update: Add replica set name to MONGO_URI env var in web/worker/web-background pods
3. Init container: Wait for replica set initialization before app starts
4. PVC per pod: Each replica gets its own PVC (WiredTiger journal + data)
5. Init job (optional): One-time job to initiate replica set with rs.initiate() and rs.add()
6. Read preference: Configure readPreference=secondaryPreferred for reporting queries
7. Backup: Document mongodump or mongodb-mongosh sidecar for periodic backups

Design Notes

• MongoDB 6.0+ replica set with PSArbiter (primary-secondary-arbiter) is standard for resource-constrained deployments
• Arbiter requires no PVC but adds the odd-numbered vote (3 total)
• WiredTiger cache size should be ~50% of available RAM per pod
• The MONGO_URI must include ?replicaSet=rs0&readPreference=secondaryPreferred for proper routing
• Phase 1 kept db-deploy.yaml unchanged since this is a Phase 2 item

References

• .agents/results/result-tf-infra.md — B2: Single MongoDB Instance (CRITICAL)
• openstudio-server/templates/db/db-deploy.yaml — current single-pod deployment
• openstudio-server/values.yaml — db.* config section

Acceptance Criteria

• 3-pod MongoDB StatefulSet with primary/secondary/arbiter roles
• Replica set auto-initializes on first deploy
• Connection string includes replica set name and read preference
• Failover to secondary completes in < 60s when primary is killed
• No data loss during failover (WiredTier journal enabled)
• Arbiter pod requires no PVC
• Backward compatible — single-pod mode still works (opt-in via values)

Start with openstudio-server/templates/db/db-deploy.yaml and the db.* section in openstudio-server/values.yaml; read .agents/results/result-tf-infra.md for the single-instance findings. Define the opt-in replica-set configuration, initialization, connection-string changes, PVC behavior, read preference, backup documentation, and failover checks against the listed acceptance criteria.