argoproj / argoproj/argo-workflows

Workflow stuck in running state with a Dag task having dependency on another task with error

Open
#14,337 1 comment 5 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

**Problem**
Workflow get stuck in running state

**How to reproduce**
Dag Task A that generate an error while evaluating an output parameter
Dag Task B having a dependency on Task A

Continue on failed and on error do not fix the issue.
Timeout and ActiveDeadlineSeconds do not stop the workflow.

It happens all the time with the sample provided.
I had the possibility to test on v3.5.5 and v3.6.2 only. The exact same issue occurs.

**Expected**
The workflow should stop

### Version(s)

v3.5.5, v3.6.2

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
metadata:
name: never-ending-workflow
namespace: argo
labels:
example: "true"
spec:
entrypoint: main
activedeadlineseconds: 30
timeout: 30
templates:
- dag:
tasks:
- name: task-a
continueOn:
error: true
template: task-a-template
timeout: 30
- name: task-b
dependencies:
- task-a
template: suspend
timeout: 30
name: main
- dag:
tasks:
- name: task-a-call-suspend
template: suspend
continueOn:
error: true
name: task-a-template
outputs:
parameters:
- name: ProblematicOutput
valueFrom:
expression: toBase64(inputs.parameters.someString)
- suspend:
duration: "1"
name: suspend
```

### Logs from the workflow controller

```text
time="2025-03-28T16:34:19.217Z" level=info msg="Processing workflow" Phase= ResourceVersion=7869 namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.227Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2025-03-28T16:34:19.228Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=never-ending-workflow3
time="2025-03-28T16:34:19.229Z" level=info msg="Updated phase -> Running" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.229Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.230Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.230Z" level=info msg="DAG node never-ending-workflow3 initialized Running" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.230Z" level=warning msg="was unable to obtain the node for never-ending-workflow3-1360456448, taskName task-b"
time="2025-03-28T16:34:19.230Z" level=warning msg="was unable to obtain the node for never-ending-workflow3-1410789305, taskName task-a"
time="2025-03-28T16:34:19.230Z" level=warning msg="was unable to obtain the node for never-ending-workflow3-1410789305, taskName task-a"
time="2025-03-28T16:34:19.230Z" level=info msg="All of node never-ending-workflow3.task-a dependencies [] completed" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.230Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.231Z" level=info msg="DAG node never-ending-workflow3-1410789305 initialized Running" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.231Z" level=warning msg="was unable to obtain the node for never-ending-workflow3-1659669732, taskName task-a-call-suspend"
time="2025-03-28T16:34:19.231Z" level=warning msg="was unable to obtain the node for never-ending-workflow3-1659669732, taskName task-a-call-suspend"
time="2025-03-28T16:34:19.231Z" level=info msg="All of node never-ending-workflow3.task-a.task-a-call-suspend dependencies [] completed" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.231Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.232Z" level=info msg="Suspend node never-ending-workflow3-1659669732 initialized Pending" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.232Z" level=info msg="node never-ending-workflow3.task-a.task-a-call-suspend suspended" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.232Z" level=info msg="node never-ending-workflow3-1659669732 phase Pending -> Running" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.232Z" level=warning msg="was unable to obtain the node for never-ending-workflow3-1360456448, taskName task-b"
time="2025-03-28T16:34:19.233Z" level=warning msg="was unable to obtain the node for never-ending-workflow3-1360456448, taskName task-b"
time="2025-03-28T16:34:19.233Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.233Z" level=info msg=reconcileAgentPod namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:19.233Z" level=info msg="Workflow to be dehydrated" Workflow Size=1766
time="2025-03-28T16:34:19.245Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=7870 workflow=never-ending-workflow3
time="2025-03-28T16:34:20.233Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=7870 namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.234Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=never-ending-workflow3
time="2025-03-28T16:34:20.235Z" level=warning msg="was unable to obtain the node for never-ending-workflow3-1360456448, taskName task-b"
time="2025-03-28T16:34:20.236Z" level=info msg="node never-ending-workflow3.task-a.task-a-call-suspend suspended" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.236Z" level=info msg="auto resuming node never-ending-workflow3.task-a.task-a-call-suspend" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.236Z" level=info msg="node never-ending-workflow3-1659669732 phase Running -> Succeeded" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.236Z" level=info msg="node never-ending-workflow3-1659669732 finished: 2025-03-28 16:34:20.236496027 +0000 UTC" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.236Z" level=error msg="unable to get outputs" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.236Z" level=error msg="Mark error node" error="unknown name inputs (1:10)\n | toBase64(inputs.parameters.someString)\n | .........^" namespace=argo nodeName=never-ending-workflow3.task-a workflow=never-ending-workflow3
time="2025-03-28T16:34:20.236Z" level=info msg="node never-ending-workflow3-1410789305 phase Running -> Error" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.236Z" level=info msg="node never-ending-workflow3-1410789305 message: unknown name inputs (1:10)\n | toBase64(inputs.parameters.someString)\n | .........^" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.236Z" level=info msg="node never-ending-workflow3-1410789305 finished: 2025-03-28 16:34:20.236853919 +0000 UTC" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.236Z" level=error msg="Mark error node" error="task 'never-ending-workflow3.task-a' errored: unknown name inputs (1:10)\n | toBase64(inputs.parameters.someString)\n | .........^" namespace=argo nodeName=never-ending-workflow3.task-a workflow=never-ending-workflow3
time="2025-03-28T16:34:20.236Z" level=info msg="node never-ending-workflow3-1410789305 message: task 'never-ending-workflow3.task-a' errored: unknown name inputs (1:10)\n | toBase64(inputs.parameters.someString)\n | .........^" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.237Z" level=warning msg="was unable to obtain the node for never-ending-workflow3-1360456448, taskName task-b"
time="2025-03-28T16:34:20.237Z" level=warning msg="was unable to obtain the node for never-ending-workflow3-1360456448, taskName task-b"
time="2025-03-28T16:34:20.237Z" level=info msg="All of node never-ending-workflow3.task-b dependencies [task-a] completed" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.237Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.238Z" level=info msg="Suspend node never-ending-workflow3-1360456448 initialized Pending" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.238Z" level=info msg="node never-ending-workflow3.task-b suspended" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.238Z" level=info msg="node never-ending-workflow3-1360456448 phase Pending -> Running" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.238Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.238Z" level=info msg=reconcileAgentPod namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:20.238Z" level=info msg="Workflow to be dehydrated" Workflow Size=2415
time="2025-03-28T16:34:20.253Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=7879 workflow=never-ending-workflow3
time="2025-03-28T16:34:21.239Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=7879 namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:21.239Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=never-ending-workflow3
time="2025-03-28T16:34:21.240Z" level=info msg="node never-ending-workflow3.task-b suspended" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:21.240Z" level=info msg="auto resuming node never-ending-workflow3.task-b" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:21.240Z" level=info msg="node never-ending-workflow3-1360456448 phase Running -> Succeeded" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:21.240Z" level=info msg="node never-ending-workflow3-1360456448 finished: 2025-03-28 16:34:21.240960077 +0000 UTC" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:21.241Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:21.241Z" level=info msg=reconcileAgentPod namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:21.241Z" level=info msg="Workflow to be dehydrated" Workflow Size=2425
time="2025-03-28T16:34:21.254Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=7884 workflow=never-ending-workflow3
time="2025-03-28T16:34:30.915Z" level=info msg="Alloc=8808 TotalAlloc=41421 Sys=22357 NumGC=19 Goroutines=185"
time="2025-03-28T16:34:31.257Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=7884 namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:31.258Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=never-ending-workflow3
time="2025-03-28T16:34:31.260Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=never-ending-workflow3
time="2025-03-28T16:34:31.260Z" level=info msg=reconcileAgentPod namespace=argo workflow=never-ending-workflow3
```

### Logs from in your workflow's wait container

```text
No resources found in argo namespace.
```

Contributor guide

Open the contributing guide

Research direction

Start by running the minimal workflow in the issue and inspect the workflow controller's DAG and task-result reconciliation, using the provided logs to follow task-a and task-b. Trace how an output-parameter evaluation error and a dependent task affect workflow completion and timeout handling. Done means the reproduced workflow reaches a terminal stopped state instead of remaining Running.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.