temporalio / temporalio/temporal-worker-controller

[Bug] Plan executor produces delete+update on same Deployment UID, then retries stale plan indefinitely

Open
#353 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

need information
Dominant language
Go
Stars
187
Forks
70
Avg merge
4d 1h
Merged PRs (30d)
31

Description

Summary

The plan executor in execplan.go can produce a reconciliation plan that contains both a "delete Deployment" and "update Deployment" step for the same k8s Deployment resource. The delete succeeds, then the update fails on a stale UID precondition. The controller retries the same plan indefinitely without recomputing from current
state.

Environment
  • Controller image: v1.5.x (exact TBD)
  • Kubernetes: EKS 1.30
  • Temporal: self-hosted (Cassandra-backed)
What happened

A helm release changed the desired image tag while an existing version was being sunset. The controller's reconcile plan included:

  1. deleting deployment for account-temporal-default-worker-17-235-1-9755
  2. deleted worker resource on version sunset (HPA) ✅
  3. updating deployment for account-temporal-default-worker-17-232-4-559b
  4. updating deployment for account-temporal-default-worker-17-235-1-9755

Step 4 failed:

StorageError: invalid object, Code: 4, Key: /registry/deployments/.../account-temporal-default-worker-17-235-1-9755,
ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: be213a24-..., UID in object meta: ""

The controller retried this exact plan for ~3 hours. Restarting the controller produced zero reconcile activity for this TWD — it didn't re-enqueue or recompute.

Impact

The Worker Deployment's current build (17.235.1-9755) had its k8s Deployment deleted but was never recreated. Zero pods existed for the current version. All new workflows routed to it were stranded.

Proposed fix
  1. Plan generator should not emit both delete and update for the same Deployment UID in a single plan.
  2. On plan execution failure, recompute the plan from current state rather than retrying the stale plan.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in execplan.go by tracing plan generation and execution for the reported delete-then-update sequence on one Deployment UID. Reproduce or test the failure path using the issue's step sequence; done means a plan does not contain both operations for the same UID and execution failure causes a fresh plan to be computed from current state.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.