temporalio / temporalio/temporal-worker-controller

[Feature Request] Version-status selector in WorkerResourceTemplate (different policy for current vs draining versions)

Open
#608 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
187
Forks
70
Avg merge
4d 1h
Merged PRs (30d)
31

Description

Is your feature request related to a problem? Please describe.

WorkerResourceTemplate renders one copy of spec.template per live versioned Deployment, with no way to say which versions it applies to. Every version gets the same policy, whether it is Current, Ramping, or Draining for days behind pinned executions.

The render loop in getWorkerResourceApplies iterates every entry in k8sState.Deployments:

for buildID, deployment := range k8sState.Deployments {

The only exclusions are Deployments already scheduled for deletion, and, when the template carries a scale target, versions in the sunset scale-down set. Version status is not consulted. WorkerResourceTemplateSpec carries only a WorkerDeployment reference and the template, and status.versions[] is keyed by Build ID alone. So one template covers Current, Ramping, and Draining alike, and the operator picks a single set of numbers that has to serve all three.

Those roles want genuinely different policy. A current version wants to scale out on backlog and wants a strict PodDisruptionBudget. A draining version has no new work by definition. It wants one poller to finish its pinned executions, and it should not hold up node drains.

KEDA / HPA. A current version may want minReplicaCount: 3, maxReplicaCount: 50 on backlog. A draining version needs exactly one poller. Today one template forces a choice: either the draining version carries the current version's floor for the whole drain, or the current version loses its floor. Our draining versions hold hundreds of workflows parked on human approval signals and stay up for days, so the difference is the entire residual cost of versioned rollouts.

PodDisruptionBudget. This one is worse than cost. The docs name PDBs as a primary use case, and spec.selector.matchLabels injection supports them per version. But a PDB written for the current version, say minAvailable: 1 or a percentage, is also rendered onto every draining version. A draining version sitting at 1 replica with minAvailable: 1 blocks eviction of its node. With a multi-day drain, that blocks node drains, cluster autoscaler consolidation, and node upgrades for as long as the version lives. The correct policy for a draining version is usually a looser PDB or none at all, and there is no way to express that.

Other kinds, same shape. VPA would want updateMode: Auto on current and Off or Initial on a draining version whose pods are idle by definition. The same split applies to the CapacityBuffer work in #509 and the WPA request in #366. This is not a property of any one resource kind. It is a gap in the WRT model, which is why a per-kind fix would not close it.

Describe the solution you'd like

Let a WorkerResourceTemplate declare the version statuses it applies to, so two WRTs against the same WorkerDeployment can express current policy and past policy separately.

Defaulting the selector to all statuses would leave existing templates unaffected. The statuses already exist as VersionStatus values, the planner already knows each version's status from status.currentVersion, status.targetVersion, and status.deprecatedVersions[], and rendered names are derived from (wdName, wrtName, buildID), so two WRTs for one WorkerDeployment produce distinct resources with no collision.

Additional context

Observed on controller v1.8.0 (kind, local Temporal server, KEDA and VPA templates), code paths re-read on v1.11.0 and main.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with getWorkerResourceApplies in internal/planner/planner.go and WorkerResourceTemplateSpec in api/v1alpha1/workerresourcetemplate_types.go. Trace how version status is derived from currentVersion, targetVersion, and deprecatedVersions[]. Done means templates can select version statuses, default existing templates to all statuses, and render distinct resources for separate templates on one WorkerDeployment.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.