temporalio / temporalio/temporal-worker-controller
[Feature Request] Argo Rollouts-like canaries (per step gate, rollback on gate failure)
@zainawaisn is already working on this.
Since Aug 19, 2026.
- Dominant language
- Go
- Stars
- 187
- Forks
- 70
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 31
Description
Is your feature request related to a problem? Please describe.
We (Descript) are looking at adopting the Temporal Worker Controller, but our existing canary workflow on Argo Rollouts depends on a few capabilities the controller doesn't yet expose.
We manage several dozen different worker deployments, which deploy three times every weekday. Manual intervention during a pause window isn't viable at that cadence. Therefore, we use Argo Rollouts to automate a 20% canary for 20 minutes in production, rolling back if the canary error rate is significantly worse than the prod error rate.
Describe the solution you'd like
There are two features we need, one that would be helpful, and one that I'll mention for the good of the community.
- Gate per step of a progressive rollout, not just at the start. The current gate feature only runs once, before any workflows have been ramped to the new build. I'm imagining a
rollout.steps[*].gatefield, structured similar torollout.gate. A step completes when its ramp percentage, pause duration, and gate (if present) have all completed (successfully). I think the simplest semantics are for the ramp change, pause timer, and gate to begin simultaneously; to gate on a stable system, multiple steps can be used. - A failed gate should rollback, not pause. The
if wf.Status != WorkflowExecutionStatusCompleted: return nilcheck means that the pods stay up until human intervention. Currently the pods are not ramped and receive no workflows, but combined with item (1), this would mean the partially ramped build continues to serve a percentage of traffic. The feature request is for a failed gate to cause the new ramping deployment to immediately drain (work in-flight completes). This can be the default for step gates, or configurable. - Proportional pre-scaling. Argo Rollouts scales pods; Temporal Worker Controller scales the percent of workflows that run on the new build. If the new build's deployment can't handle the work, work queues until it scales, which takes a few minutes. We try to operate with empty queues to minimize latency. The solution, previously proposed in issue #14, is to preemptively scale up the new build's deployment, proportionally to the size of the ramp. The pause timer (and gate from item 2) would begin only once the scale up is complete; some timeout on the scale up would catch unhealthy pods and cause the deployment to drain. We could work around the absence of this feature with a suitably gradual ramp (like 1% a minute), but that adds latency to deploys and the config is ugly.
- There is no automatic way to pass which workflow is being analyzed by the gate workflow. A single healthcheck workflow parameterized over which workflow to check seems like the clear solution. This is easy for us to work around because we generate our k8s manifests, but mentioning it in case it's useful for other users.
Additional context
We'd be happy to draft proposals or contribute PRs of these features, but I wanted to confirm that the proposed direction and sketched implementation match what the project maintainers have in mind.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.