Resource-Aware Worker Concurrency

Problem

Today, Temporal workers enforce concurrency with static limits (max_concurrent_activity_task_executions, max_concurrent_workflow_task_executions, etc.). These limits assume that workloads are uniform, but in practice workflows and activities have very different profiles (short IO-bound tasks vs. long CPU-bound tasks vs. memory-heavy ones).

This creates a false trade-off:

• Set limits too low → workers are underutilized, leaving CPU/RAM idle and wasting money.
• Set limits too high → workers oversubscribe, pods risk OOM/CPU thrash, and robustness is undermined.

Kubernetes autoscaling works on CPU/memory thresholds, but Temporal workers don’t adapt to that. The two systems don’t “speak the same language,” leading to either inefficiency or instability.

Proposed Direction

Introduce resource-aware concurrency controls in workers. For example:

• Workers should stop polling when pod CPU/memory reaches configurable thresholds (e.g. 70–80%).
• Concurrency could scale dynamically based on resource availability, rather than fixed magic numbers.
• Optionally, expose hooks so operators can plug in their own resource metrics or scaling policies.

This would allow Temporal’s built-in backpressure to align with Kubernetes autoscaling, ensuring:

• Pods never overload themselves.
• Idle capacity is minimized.
• Scaling is smooth and cost-efficient.

Why This Matters

Temporal’s core value proposition is reliability at scale. Without resource-aware workers, teams are forced to choose between:

• Wasted resources (cost inefficiency).
• Risky oversubscription (reliability gaps).

A resource-adaptive model would make Temporal more robust, cloud-native, and cost-efficient out of the box.

No files, tests, or implementation entry points are named. Start by locating the worker concurrency limits and polling/backpressure behavior, then determine how configurable CPU and memory thresholds or operator-provided metrics should interact with them; done requires an agreed design and tests covering resource-aware polling or concurrency changes.