Phase 2: Queue Priority Mechanism for Fair Scheduling

Problem

Resque uses FIFO ordering on a single simulations list. There is no priority queuing, backpressure mechanism, or job batching (B11). At scale, requeued jobs (retries) compete with fresh jobs at the same priority, and long-running simulations can block shorter ones.

What Phase 1 Did

• Nothing — queue structure unchanged. worker.queues: "simulations,requeued" still uses flat priority.

What Should Be Done (This Issue)

1. Multiple queue support: Add ability to configure separate Resque queues for different priority levels (e.g., expedited, bulk, requeued)
2. Queue routing: Allow web-background to enqueue jobs to different queues based on analysis parameters (requires app changes)
3. Worker queue configuration: Support worker.queueGroups in values.yaml, each with its own worker count, queue list, and scaling behavior
4. Separate worker deployments (optional): Create dedicated worker Deployment(s) for high-priority queues with different resource profiles
5. Backpressure: Add configurable queue depth limit — when simulations queue exceeds threshold, web-background stops creating new jobs until it drains
6. Dashboard: Add Resque queue depth annotation to Prometheus metrics

Design Notes

• Kubernetes-level approach: multiple worker Deployments with different QUEUES values
• Requires OpenStudio Server application changes to route jobs to correct queues
• Resque naturally supports multiple queues — QUEUES=expedited,simulations,requeued processes in priority order
• Backpressure is best implemented at the app layer (web-background checks queue depth before enqueueing)
• The chart should support worker.additionalDeployments list for custom queue groups

References

• .agents/results/result-tf-infra.md — B11: FIFO Queue Fairness (MEDIUM)
• openstudio-server/values.yaml — worker.queues
• openstudio-server/templates/worker/worker-deploy.yaml

Acceptance Criteria

• worker.queueGroups config creates separate Deployments per queue group
• Expedited jobs bypass bulk queue (app-level change documented)
• Queue depth limit prevents unbounded backlog
• Prometheus metrics expose per-queue depth
• Backward compatible — existing single-queue config still works

Start by reading openstudio-server/values.yaml and openstudio-server/templates/worker/worker-deploy.yaml, then trace how worker.queues is rendered into deployments. The work is done when queueGroups can create separate worker Deployments while preserving the existing configuration, with the documented application-level routing, queue limits, and per-queue metrics covered by the acceptance criteria.