argoproj / argoproj/argo-workflows
Inconsistent workflow and entryNode phase due to persistence db instability
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
We encountered a workflow error caused by database instability.
However, what’s strange is that its entryNode phase still shows as `Running`, even though the workflow has already `Error`.
This makes the tree view in the argo-workflows ui look like it’s still running — and it never gets fixed.
```
metadata:
name: my-workflow-1758067200
spec: {...}
status:
phase: Error <-- here
startedAt: "2025-09-17T00:00:00Z"
finishedAt: "2025-09-17T17:55:42Z"
estimatedDuration: 81284
progress: 0/0
message: "FATAL: terminating connection due to administrator command (SQLSTATE 57P01)"
nodes:
my-workflow-1758067200:
id: my-workflow-1758067200
name: my-workflow-1758067200
displayName: my-workflow-1758067200
type: Steps
templateName: main
templateScope: local/
phase: Running <-- here
startedAt: "2025-09-17T00:00:00Z"
finishedAt: null
progress: 1031/1033
```
After investigation, we found that when a controller tries to hydrate a workflow from persistence database (offload nodes) and an error occurs,
the controller attempts to mark both the workflow (snippet 1) and the entryNode as `Error` (snippet 2).
However, since the hydration didn’t succeed, `.status.nodes.${entry_node}` doesn’t exist at that point — so the controller can’t update its phase (snippet 2).
As a result, the workflow moves on to the pending archive stage, and when it’s finally archived (assume the db is back to normal), the entryNode’s phase remains stuck in its previous state (`Running`).
snippet 1
https://github.com/argoproj/argo-workflows/blob/741ab0ef7b6432925e49882cb4294adccf5912ec/workflow/controller/controller.go#L891-L897
snippet 2
https://github.com/argoproj/argo-workflows/blob/741ab0ef7b6432925e49882cb4294adccf5912ec/workflow/controller/operator.go#L2381-L2392
I think this issue is difficult to be completely fixed, but if we could re-check the phase of the workflow and the entryNode before archiving it, it could at least reduce the confusion caused by inconsistent phases on ui.
or do you have a better suggestion?
### Version(s)
v3.6.2
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
```YAML
N/A
```
### Logs from the workflow controller
```text
2025-09-18 01:55:42.400 (2025-09-17T17:55:42.400Z)
level=error msg=\""hydration failed: FATAL: terminating connection due to administrator command (SQLSTATE 57P01)\"" namespace=argo workflow=my-workflow-1758067200""}"
level=info msg=\""Updated phase Running -> Error\"" namespace=argo workflow=my-workflow-1758067200""}"
level=info msg=\""Updated message -> FATAL: terminating connection due to administrator command (SQLSTATE 57P01)\"" namespace=argo workflow=my-workflow-1758067200""}"
level=error msg=\""was unable to obtain node for my-workflow-1758067200\"" namespace=argo workflow=my-workflow-1758067200""}"
level=info msg=\""Marking workflow completed\"" namespace=argo workflow=my-workflow-1758067200""}"
level=info msg=\""Marking workflow as pending archiving\"" namespace=argo workflow=my-workflow-1758067200""}"
```
### Logs from in your workflow's wait container
```text
N/A
```
Contributor guide
Research direction
Start by reading the referenced sections of workflow/controller/controller.go and workflow/controller/operator.go, then trace the hydration-failure path through completion and pending archiving. The fix should prevent a workflow from being archived with its entryNode still Running after the workflow becomes Error, and should preserve consistent phases in the UI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100