temporalio / temporalio/temporal
Worker Deployment version GC does not reclaim eligible drained versions at maxVersionsInDeployment, wedging rollouts
@Shivs11 is already working on this.
Since Aug 4, 2026.
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
What are you trying to do
Run Worker Versioning (Worker Deployments) with a controller that deploys frequently. Each deploy registers a new Worker Deployment Version. We expect old, drained versions to be reclaimed so we never hit the matching.maxVersionsInDeployment cap.
Describe the bug
Drained Worker Deployment Version records are not reclaimed in practice, so a deployment's version count grows by one per deploy until it reaches the cap, after which a new build can no longer register as a poller and the rollout is stuck (the new version stays unregistered and cannot be promoted to current).
Observations on a busy namespace (server 1.30.x; also reproduced by another user on 1.29.1, see the forum link below):
- A single Worker Deployment had accumulated well over 100 versions. The large majority were
drainedand had no pollers (their workers were scaled to zero long ago). The oldest drained version dated back to the deployment's creation months earlier and had never been reclaimed. - When the deployment reached the configured cap, registering the next version failed with
cannot add version ... since maximum number of versions (N) have been registered in the deployment, even though many of the existing versions were drained with no pollers and should have been eligible for the server's at-cap reclaim. - Manually deleting drained versions via
DeleteWorkerDeploymentVersionimmediately unblocked registration.
So two things appear to be happening:
- Below the cap there is no background reclamation of drained, poller-less versions. They accumulate up to the cap.
- At the cap, the lazy reclaim that is supposed to delete the oldest eligible version does not free room, despite many drained, poller-less versions being present.
Expected behavior
Either the oldest drained, poller-less version is reliably reclaimed when a new version is registered at the cap (the documented behavior), or drained, poller-less versions are reclaimed in the background before the cap is reached. A deployment doing frequent deploys should not wedge.
Questions
- At the cap, does the reclaim path iterate from oldest to newest to find any eligible version (drained, no pollers, not current or ramping), or does it only try the single oldest and give up if that one is ineligible? Our repro suggests eligible versions were present but not reclaimed.
- What is the exact eligibility predicate for at-cap deletion, and how long after a version drains does it become eligible?
- Is there any background reclamation below the cap, or is reclamation only ever triggered by registering a new version at the cap?
Versions and Tags
Server 1.30.x (reproduced on 1.29.1 per the forum thread).
Additional context
Community forum thread describing the same symptom, currently unanswered: https://community.temporal.io/t/worker-versioning-garbage-collection-bug/19600
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.