[BUG] v2: run absent from UI but its TaskAction CR reconciles forever, recreating the task pod
- Dominant language
- Go
- Stars
- 7.5k
- Forks
- 886
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 120
Description
## Describe the bug
A v2 run entered a state where the control plane, the backend worker, and the TaskAction CRs disagreed about whether it exists:
- The run no longer appeared as running in the console, so there was no way to abort it from the UI.
- Its root TaskAction CR (`-a0`, `flyte.org/is-root: "true"`) remained in the cluster with condition `Progressing / reason=Executing` for **17 days** after creation, never reaching a terminal phase and never being GC'd. Its status phase history cycles `Queued → Initializing → Executing` repeatedly.
- The taskaction-controller kept the task pod alive indefinitely: deleting the pod (exit 137) just had it recreated within seconds.
- Meanwhile the backend worker was stuck in a retry loop on a sibling action of the same run:
```
[flyte] ERROR Failed to launch action: , Code: Code.INTERNAL, Details failed to create TaskAction CR -: taskactions.flyte.org "-" already exists backing off...
[flyte] WARNING [worker-0] Backing off for 10.0 [retry 6/100] on action due to error: ...
[][a0] [flyte] WARNING Informer watch retrying, attempt 4/40
[][a0] [flyte] ERROR RPC error: .a0, Error reading content: request or response body error: error reading a body from connection: unexpected EOF during chunk size line
```
and the controller's events were rejected:
```
ERROR events Server rejected event (will not retry!) ... Reason:FailedPluginHandle ... Note:Plugin "pod" system error: [Interrupted] Pod failed. No message received from kubernetes.
[-a0-0] terminated with exit code (137). Reason [Error].
```
The practical impact for us: the orphaned pod carried `karpenter.sh/do-not-disrupt`, so it blocked a node drain for hours until we manually `kubectl delete taskactions -l flyte.org/run=` (which also stopped the worker's retry loop).
## Expected behavior
When a run is terminal (or no longer known to the control plane), its TaskAction CRs should converge to a terminal phase and be garbage-collected, and the controller should stop recreating pods. At minimum, a run in this state should be visible/abortable so the CRs can be cleaned up through the API rather than kubectl.
## Steps to reproduce
Unknown trigger — the run was launched from a schedule and the desync was noticed 17 days later. The informer-watch errors in the logs suggest a dropped watch may have started it. Happy to provide the full TaskAction/pod YAML and larger log excerpts.
## Environment
- Backend: `ghcr.io/flyteorg/flyte-binary-v2:v2.0.42` (flyte-binary v2 chart)
- Kubernetes: EKS v1.36
Contributor guide
Research direction
Start by inspecting the TaskAction CR and pod YAML, then trace the taskaction-controller, backend worker retry loop, and informer-watch errors described in the logs. Compare the root and sibling actions across their phase histories and events. Done means the failure can be reproduced or explained and orphaned actions converge to a terminal phase without recreating pods.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100