argoproj / argoproj/argo-workflows

Aggregated result of loop is not resolved when empty array

Open
#13,193 0 comments 0 reactions 0 assignees View on GitHub
area/looping area/templating P3
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [X] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened/what did you expect to happen?

The value resolved to `[]`, but instead I got the template value:
```
{{steps.process-array.outputs.result}}
```

Which is not the case where the previous step was running (e.g. the original array has more then 1 item)

### Version

latest

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: three-step-workflow-
spec:
entrypoint: main
serviceAccountName: argo
templates:
- name: main
steps:
- - name: generate-array
template: generate-array
- - name: process-array
template: process-array
arguments:
parameters:
- name: element
value: "{{item}}"
withParam: "{{steps.generate-array.outputs.result}}"
- - name: final-step
template: final-step
arguments:
parameters:
- name: processed-results
value: "{{steps.process-array.outputs.result}}"

- name: generate-array
script:
image: python:3.8
command: [python]
source: |
import json
array = []
print(json.dumps(array))

- name: process-array
inputs:
parameters:
- name: element
script:
image: python:3.8
command: [python]
source: |
import json
element = "{{inputs.parameters.element}}"
result = f"processed-{element}"
print(json.dumps(result))

- name: final-step
inputs:
parameters:
- name: processed-results
script:
image: python:3.8
command: [python]
source: |
import json
processed_results = json.loads('{{inputs.parameters.processed-results}}')
final_result = " ".join(processed_results)
print(final_result)
```

### Logs from the workflow controller

```text
time="2024-06-16T05:13:55.081Z" level=info msg="Processing workflow" Phase= ResourceVersion=1587 namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:13:55.086Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2024-06-16T05:13:55.086Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2024-06-16T05:13:55.086Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=three-step-workflow-mp5xb
time="2024-06-16T05:13:55.086Z" level=info msg="Updated phase -> Running" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:13:55.086Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:13:55.087Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:13:55.087Z" level=info msg="Steps node three-step-workflow-mp5xb initialized Running" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:13:55.087Z" level=info msg="StepGroup node three-step-workflow-mp5xb-2582394757 initialized Running" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:13:55.087Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:13:55.087Z" level=info msg="Pod node three-step-workflow-mp5xb-2804180898 initialized Pending" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:13:55.098Z" level=info msg="Created pod: three-step-workflow-mp5xb[0].generate-array (three-step-workflow-mp5xb-generate-array-2804180898)" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:13:55.098Z" level=info msg="Workflow step group node three-step-workflow-mp5xb-2582394757 not yet completed" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:13:55.098Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:13:55.098Z" level=info msg=reconcileAgentPod namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:13:55.099Z" level=info msg="Workflow to be dehydrated" Workflow Size=2116
time="2024-06-16T05:13:55.109Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1591 workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.099Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1591 namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.099Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.099Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/latest/workflow-rbac/" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.099Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/latest/workflow-rbac/" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=three-step-workflow-mp5xb-2804180898 old.message= old.phase=Pending old.progress=0/1 workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=info msg="Step group node three-step-workflow-mp5xb-2582394757 successful" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=info msg="node three-step-workflow-mp5xb-2582394757 phase Running -> Succeeded" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=info msg="node three-step-workflow-mp5xb-2582394757 finished: 2024-06-16 05:14:05.100303755 +0000 UTC" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=info msg="StepGroup node three-step-workflow-mp5xb-3589198992 initialized Running" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=info msg="SG Outbound nodes of three-step-workflow-mp5xb-2804180898 are [three-step-workflow-mp5xb-2804180898]" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=info msg="Skipping three-step-workflow-mp5xb[1].process-array: Skipped, empty params" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=info msg="Skipped node three-step-workflow-mp5xb-778085927 initialized Skipped (message: Skipped, empty params)" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=info msg="Step group node three-step-workflow-mp5xb-3589198992 successful" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=info msg="node three-step-workflow-mp5xb-3589198992 phase Running -> Succeeded" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=info msg="node three-step-workflow-mp5xb-3589198992 finished: 2024-06-16 05:14:05.100522421 +0000 UTC" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=info msg="StepGroup node three-step-workflow-mp5xb-3588757707 initialized Running" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=info msg="SG Outbound nodes of three-step-workflow-mp5xb-778085927 are [three-step-workflow-mp5xb-778085927]" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.100Z" level=info msg="Pod node three-step-workflow-mp5xb-629760346 initialized Pending" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.106Z" level=info msg="Created pod: three-step-workflow-mp5xb[2].final-step (three-step-workflow-mp5xb-final-step-629760346)" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.106Z" level=info msg="Workflow step group node three-step-workflow-mp5xb-3588757707 not yet completed" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.106Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.106Z" level=info msg=reconcileAgentPod namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.107Z" level=info msg="Workflow to be dehydrated" Workflow Size=3817
time="2024-06-16T05:14:05.112Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1631 workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:05.118Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/three-step-workflow-mp5xb-generate-array-2804180898/labelPodCompleted
time="2024-06-16T05:14:15.109Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1631 namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.109Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.109Z" level=info msg="Pod failed: Error (exit code 1)" displayName=final-step namespace=argo pod=three-step-workflow-mp5xb-final-step-629760346 templateName=final-step workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.109Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/latest/workflow-rbac/" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="node changed" namespace=argo new.message="Error (exit code 1)" new.phase=Failed new.progress=0/1 nodeID=three-step-workflow-mp5xb-629760346 old.message= old.phase=Pending old.progress=0/1 workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="SG Outbound nodes of three-step-workflow-mp5xb-2804180898 are [three-step-workflow-mp5xb-2804180898]" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="SG Outbound nodes of three-step-workflow-mp5xb-778085927 are [three-step-workflow-mp5xb-778085927]" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="Step group node three-step-workflow-mp5xb-3588757707 deemed failed: child 'three-step-workflow-mp5xb-629760346' failed" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="node three-step-workflow-mp5xb-3588757707 phase Running -> Failed" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="node three-step-workflow-mp5xb-3588757707 message: child 'three-step-workflow-mp5xb-629760346' failed" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="node three-step-workflow-mp5xb-3588757707 finished: 2024-06-16 05:14:15.110789384 +0000 UTC" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="step group three-step-workflow-mp5xb-3588757707 was unsuccessful: child 'three-step-workflow-mp5xb-629760346' failed" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="Outbound nodes of three-step-workflow-mp5xb-629760346 is [three-step-workflow-mp5xb-629760346]" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="Outbound nodes of three-step-workflow-mp5xb is [three-step-workflow-mp5xb-629760346]" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="node three-step-workflow-mp5xb phase Running -> Failed" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="node three-step-workflow-mp5xb message: child 'three-step-workflow-mp5xb-629760346' failed" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="node three-step-workflow-mp5xb finished: 2024-06-16 05:14:15.110846301 +0000 UTC" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg=reconcileAgentPod namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="Updated phase Running -> Failed" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.110Z" level=info msg="Updated message -> child 'three-step-workflow-mp5xb-629760346' failed" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.111Z" level=info msg="Marking workflow completed" namespace=argo workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.111Z" level=info msg="Workflow to be dehydrated" Workflow Size=4251
time="2024-06-16T05:14:15.116Z" level=info msg="Workflow update successful" namespace=argo phase=Failed resourceVersion=1673 workflow=three-step-workflow-mp5xb
time="2024-06-16T05:14:15.116Z" level=info msg="cleaning up pod" action=deletePod key=argo/three-step-workflow-mp5xb-1340600742-agent/deletePod
time="2024-06-16T05:14:15.118Z" level=warning msg="failed to clean-up pod" action=deletePod error="pods \"three-step-workflow-mp5xb-1340600742-agent\" not found" key=argo/three-step-workflow-mp5xb-1340600742-agent/deletePod
time="2024-06-16T05:14:15.118Z" level=warning msg="Non-transient error: pods \"three-step-workflow-mp5xb-1340600742-agent\" not found"
time="2024-06-16T05:14:15.131Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/three-step-workflow-mp5xb-final-step-629760346/labelPodCompleted
time="2024-06-16T05:15:47.431Z" level=info msg="Alloc=5935 TotalAlloc=24051 Sys=28773 NumGC=8 Goroutines=175"
```

### Logs from in your workflow's wait container

```text
time="2024-06-16T05:14:07.928Z" level=info msg="Starting deadline monitor"
time="2024-06-16T05:14:09.930Z" level=info msg="Main container completed" error=""
time="2024-06-16T05:14:09.930Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-06-16T05:14:09.930Z" level=info msg="No output parameters"
time="2024-06-16T05:14:09.930Z" level=info msg="No output artifacts"
time="2024-06-16T05:14:09.942Z" level=warning msg="failed to patch task result, falling back to legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/latest/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:argo:argo\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2024-06-16T05:14:09.958Z" level=info msg="Alloc=7145 TotalAlloc=12854 Sys=20069 NumGC=4 Goroutines=8"
time="2024-06-16T05:14:09.959Z" level=warning msg="failed to patch task result, falling back to legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/latest/workflow-rbac/" error="workflowtaskresults.argoproj.io \"three-step-workflow-mp5xb-629760346\" is forbidden: User \"system:serviceaccount:argo:argo\" cannot patch resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2024-06-16T05:14:09.966Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2024-06-16T05:14:09.966Z" level=info msg="Deadline monitor stopped"
time="2024-06-16T05:13:57.714Z" level=info msg="Starting deadline monitor"
time="2024-06-16T05:13:59.721Z" level=info msg="Main container completed" error=""
time="2024-06-16T05:13:59.721Z" level=info msg="Capturing script output"
time="2024-06-16T05:13:59.721Z" level=info msg="No output parameters"
time="2024-06-16T05:13:59.721Z" level=info msg="No output artifacts"
time="2024-06-16T05:13:59.744Z" level=warning msg="failed to patch task result, falling back to legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/latest/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:argo:argo\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2024-06-16T05:13:59.767Z" level=info msg="Alloc=7380 TotalAlloc=12908 Sys=24421 NumGC=4 Goroutines=8"
time="2024-06-16T05:13:59.769Z" level=warning msg="failed to patch task result, falling back to legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/latest/workflow-rbac/" error="workflowtaskresults.argoproj.io \"three-step-workflow-mp5xb-2804180898\" is forbidden: User \"system:serviceaccount:argo:argo\" cannot patch resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2024-06-16T05:13:59.782Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2024-06-16T05:13:59.782Z" level=info msg="Deadline monitor stopped"
```

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied three-step workflow and inspect the workflow controller logs when generate-array returns an empty array and process-array is skipped. Trace how the final-step parameter is resolved, and consider the issue done when it receives an empty aggregated result instead of the unresolved template expression.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.