argoproj / argoproj/argo-workflows

Inconsistent workflow and entryNode phase due to persistence db instability

Open
#14,907 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

We encountered a workflow error caused by database instability.
However, what’s strange is that its entryNode phase still shows as `Running`, even though the workflow has already `Error`.
This makes the tree view in the argo-workflows ui look like it’s still running — and it never gets fixed.

```
metadata:
name: my-workflow-1758067200
spec: {...}
status:
phase: Error <-- here
startedAt: "2025-09-17T00:00:00Z"
finishedAt: "2025-09-17T17:55:42Z"
estimatedDuration: 81284
progress: 0/0
message: "FATAL: terminating connection due to administrator command (SQLSTATE 57P01)"
nodes:
my-workflow-1758067200:
id: my-workflow-1758067200
name: my-workflow-1758067200
displayName: my-workflow-1758067200
type: Steps
templateName: main
templateScope: local/
phase: Running <-- here
startedAt: "2025-09-17T00:00:00Z"
finishedAt: null
progress: 1031/1033
```

After investigation, we found that when a controller tries to hydrate a workflow from persistence database (offload nodes) and an error occurs,
the controller attempts to mark both the workflow (snippet 1) and the entryNode as `Error` (snippet 2).

However, since the hydration didn’t succeed, `.status.nodes.${entry_node}` doesn’t exist at that point — so the controller can’t update its phase (snippet 2).
As a result, the workflow moves on to the pending archive stage, and when it’s finally archived (assume the db is back to normal), the entryNode’s phase remains stuck in its previous state (`Running`).

snippet 1
https://github.com/argoproj/argo-workflows/blob/741ab0ef7b6432925e49882cb4294adccf5912ec/workflow/controller/controller.go#L891-L897

snippet 2
https://github.com/argoproj/argo-workflows/blob/741ab0ef7b6432925e49882cb4294adccf5912ec/workflow/controller/operator.go#L2381-L2392

I think this issue is difficult to be completely fixed, but if we could re-check the phase of the workflow and the entryNode before archiving it, it could at least reduce the confusion caused by inconsistent phases on ui.
or do you have a better suggestion?

### Version(s)

v3.6.2

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
N/A
```

### Logs from the workflow controller

```text
2025-09-18 01:55:42.400 (2025-09-17T17:55:42.400Z)
level=error msg=\""hydration failed: FATAL: terminating connection due to administrator command (SQLSTATE 57P01)\"" namespace=argo workflow=my-workflow-1758067200""}"
level=info msg=\""Updated phase Running -> Error\"" namespace=argo workflow=my-workflow-1758067200""}"
level=info msg=\""Updated message -> FATAL: terminating connection due to administrator command (SQLSTATE 57P01)\"" namespace=argo workflow=my-workflow-1758067200""}"
level=error msg=\""was unable to obtain node for my-workflow-1758067200\"" namespace=argo workflow=my-workflow-1758067200""}"
level=info msg=\""Marking workflow completed\"" namespace=argo workflow=my-workflow-1758067200""}"
level=info msg=\""Marking workflow as pending archiving\"" namespace=argo workflow=my-workflow-1758067200""}"
```

### Logs from in your workflow's wait container

```text
N/A
```

Contributor guide

Open the contributing guide

Research direction

Start by reading the referenced sections of workflow/controller/controller.go and workflow/controller/operator.go, then trace the hydration-failure path through completion and pending archiving. The fix should prevent a workflow from being archived with its entryNode still Running after the workflow becomes Error, and should preserve consistent phases in the UI.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.