temporalio / temporalio/temporal-worker-controller

[Feature Request] Argo Rollouts-like canaries (per step gate, rollback on gate failure)

Open
#365 1 comment 0 reactions 1 assignee View on GitHub

@zainawaisn is already working on this.

Since Aug 19, 2026.

Dominant language
Go
Stars
187
Forks
70
Avg merge
4d 1h
Merged PRs (30d)
31

Description

Is your feature request related to a problem? Please describe.

We (Descript) are looking at adopting the Temporal Worker Controller, but our existing canary workflow on Argo Rollouts depends on a few capabilities the controller doesn't yet expose.

We manage several dozen different worker deployments, which deploy three times every weekday. Manual intervention during a pause window isn't viable at that cadence. Therefore, we use Argo Rollouts to automate a 20% canary for 20 minutes in production, rolling back if the canary error rate is significantly worse than the prod error rate.

Describe the solution you'd like

There are two features we need, one that would be helpful, and one that I'll mention for the good of the community.

  1. Gate per step of a progressive rollout, not just at the start. The current gate feature only runs once, before any workflows have been ramped to the new build. I'm imagining a rollout.steps[*].gate field, structured similar to rollout.gate. A step completes when its ramp percentage, pause duration, and gate (if present) have all completed (successfully). I think the simplest semantics are for the ramp change, pause timer, and gate to begin simultaneously; to gate on a stable system, multiple steps can be used.
  2. A failed gate should rollback, not pause. The if wf.Status != WorkflowExecutionStatusCompleted: return nil check means that the pods stay up until human intervention. Currently the pods are not ramped and receive no workflows, but combined with item (1), this would mean the partially ramped build continues to serve a percentage of traffic. The feature request is for a failed gate to cause the new ramping deployment to immediately drain (work in-flight completes). This can be the default for step gates, or configurable.
  3. Proportional pre-scaling. Argo Rollouts scales pods; Temporal Worker Controller scales the percent of workflows that run on the new build. If the new build's deployment can't handle the work, work queues until it scales, which takes a few minutes. We try to operate with empty queues to minimize latency. The solution, previously proposed in issue #14, is to preemptively scale up the new build's deployment, proportionally to the size of the ramp. The pause timer (and gate from item 2) would begin only once the scale up is complete; some timeout on the scale up would catch unhealthy pods and cause the deployment to drain. We could work around the absence of this feature with a suitably gradual ramp (like 1% a minute), but that adds latency to deploys and the config is ugly.
  4. There is no automatic way to pass which workflow is being analyzed by the gate workflow. A single healthcheck workflow parameterized over which workflow to check seems like the clear solution. This is easy for us to work around because we generate our k8s manifests, but mentioning it in case it's useful for other users.
Additional context

We'd be happy to draft proposals or contribute PRs of these features, but I wanted to confirm that the proposed direction and sketched implementation match what the project maintainers have in mind.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.