temporalio / temporalio/temporal-worker-controller
[Bug] Plan executor produces delete+update on same Deployment UID, then retries stale plan indefinitely
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 187
- Forks
- 70
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 31
Description
Summary
The plan executor in execplan.go can produce a reconciliation plan that contains both a "delete Deployment" and "update Deployment" step for the same k8s Deployment resource. The delete succeeds, then the update fails on a stale UID precondition. The controller retries the same plan indefinitely without recomputing from current
state.
Environment
- Controller image: v1.5.x (exact TBD)
- Kubernetes: EKS 1.30
- Temporal: self-hosted (Cassandra-backed)
What happened
A helm release changed the desired image tag while an existing version was being sunset. The controller's reconcile plan included:
deleting deploymentforaccount-temporal-default-worker-17-235-1-9755✅deleted worker resource on version sunset(HPA) ✅updating deploymentforaccount-temporal-default-worker-17-232-4-559b✅updating deploymentforaccount-temporal-default-worker-17-235-1-9755❌
Step 4 failed:
StorageError: invalid object, Code: 4, Key: /registry/deployments/.../account-temporal-default-worker-17-235-1-9755,
ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: be213a24-..., UID in object meta: ""
The controller retried this exact plan for ~3 hours. Restarting the controller produced zero reconcile activity for this TWD — it didn't re-enqueue or recompute.
Impact
The Worker Deployment's current build (17.235.1-9755) had its k8s Deployment deleted but was never recreated. Zero pods existed for the current version. All new workflows routed to it were stranded.
Proposed fix
- Plan generator should not emit both delete and update for the same Deployment UID in a single plan.
- On plan execution failure, recompute the plan from current state rather than retrying the stale plan.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in execplan.go by tracing plan generation and execution for the reported delete-then-update sequence on one Deployment UID. Reproduce or test the failure path using the issue's step sequence; done means a plan does not contain both operations for the same UID and execution failure causes a fresh plan to be computed from current state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100