argoproj / argoproj/argo-workflows

Multiple references to consecutive withParam in same workflow will fail.

Open
#9,474 13 comments 8 reactions 1 assignee Claimed by @isubasinghe View on GitHub
area/looping area/templating P1
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

## Checklist

* [x] Double-checked my configuration.
* [x] Tested using the latest version.
* [x] Used the Emissary executor.

## Summary

When using withParam to read values from a json file written by previous steps, the workflow will fail to parse the json file upon the 3rd consecutive reference to withParam.

To emphasize the issue, i have written a test workflow with the 2nd task calling withParam (Task C) but not actually relying on any values from it. This will cause the subsequent task (Task D) to fail at parsing its json input, but if withParam is commented out in task C the subsequent task D will parse the json input just fine.

What version are you running?
v3.3.9
## Diagnostics

Paste the smallest workflow that reproduces the bug. We must be able to run the workflow.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: dag-diamond-jsontest-
annotations:
workflows.argoproj.io/description: |
This is an example of of breaking withParam.
spec:
serviceAccountName: workflow
entrypoint: diamond
templates:
- name: diamond
dag:
tasks:
- name: A
template: writeJson
arguments:
parameters:
- name: dummyv1
value: "dv1"
- name: dummyv2
value: "dv2"
- name: B
arguments:
parameters:
- name: dummyv1
value: "tmp"
- name: dummyv2
value: "tmp2"
- name: jsonOut
value: "{{item}}"
withParam: "{{tasks.A.outputs.parameters.stepout}}"
depends: "A"
template: writeJson
- name: C
arguments:
parameters:
- name: dummyv1
value: "tmp"
- name: dummyv2
value: "tmp2"
- name: jsonOut
value: "{{item}}"
withParam: "{{tasks.B.outputs.parameters.stepout}}"
depends: "B"
template: writeJson
- name: D
arguments:
parameters:
- name: dummyv1
value: "{{item.dummyv1}}"
- name: dummyv2
value: "{{item.dummyv2}}"
withParam: "{{tasks.C.outputs.parameters.stepout}}"
depends: "B && C"
template: writeJson

- name: writeJson
inputs:
parameters:
- name: dummyv1
- name: dummyv2
script:
image: python:alpine3.6
command: [python]
source: |
import json
args = [dict(dummyv1='one', dummyv2='two')]
with open("/tmp/myfile.json", "w") as fd:
tmp = json.dumps(args)
fd.write(tmp)
outputs:
parameters:
- name: stepout
valueFrom:
path: /tmp/myfile.json
```
```bash
STEP TEMPLATE PODNAME DURATION MESSAGE
⚠ dag-diamond-jsontest-kf7td diamond
├─✔ A writeJson dag-diamond-jsontest-kf7td-1806111301 3s
├─✔ B(0:dummyv1:one,dummyv2:two) writeJson dag-diamond-jsontest-kf7td-3988660372 3s
├─✔ C(0:[{\"dummyv1\": \"one\", \"dummyv2\": \"two\"}]) writeJson dag-diamond-jsontest-kf7td-913068289 3s
└─⚠ D writeJson failed to resolve {{item.dummyv1}}
```

```bash
# Logs from the workflow controller:
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
time="2022-08-30T23:35:01.305Z" level=info msg="Processing workflow" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.309Z" level=info msg="Updated phase -> Running" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.309Z" level=info msg="DAG node dag-diamond-jsontest-kf7td initialized Running" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.309Z" level=info msg="All of node dag-diamond-jsontest-kf7td.A dependencies [] completed" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.309Z" level=info msg="Pod node dag-diamond-jsontest-kf7td-1806111301 initialized Pending" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.314Z" level=info msg="Created pod: dag-diamond-jsontest-kf7td.A (dag-diamond-jsontest-kf7td-1806111301)" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.314Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.314Z" level=info msg=reconcileAgentPod namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:01.318Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=6712 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.316Z" level=info msg="Processing workflow" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.316Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.316Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.316Z" level=info msg="node changed" new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-diamond-jsontest-kf7td-1806111301 old.message= old.phase=Pending old.progress=0/1
time="2022-08-30T23:35:11.318Z" level=info msg="TaskGroup node dag-diamond-jsontest-kf7td-1755778444 initialized Running (message: )" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.318Z" level=info msg="All of node dag-diamond-jsontest-kf7td.B(0:dummyv1:one,dummyv2:two) dependencies [A] completed" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.318Z" level=info msg="Pod node dag-diamond-jsontest-kf7td-3988660372 initialized Pending" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.327Z" level=info msg="Created pod: dag-diamond-jsontest-kf7td.B(0:dummyv1:one,dummyv2:two) (dag-diamond-jsontest-kf7td-3988660372)" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.327Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.327Z" level=info msg=reconcileAgentPod namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.330Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=6744 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:11.336Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/dag-diamond-jsontest-kf7td-1806111301/labelPodCompleted
time="2022-08-30T23:35:21.329Z" level=info msg="Processing workflow" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.329Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.329Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.330Z" level=info msg="node changed" new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-diamond-jsontest-kf7td-3988660372 old.message= old.phase=Pending old.progress=0/1
time="2022-08-30T23:35:21.331Z" level=info msg="node dag-diamond-jsontest-kf7td-1755778444 phase Running -> Succeeded" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.331Z" level=info msg="node dag-diamond-jsontest-kf7td-1755778444 finished: 2022-08-30 23:35:21.33152398 +0000 UTC" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.332Z" level=info msg="TaskGroup node dag-diamond-jsontest-kf7td-1772556063 initialized Running (message: )" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.332Z" level=info msg="All of node dag-diamond-jsontest-kf7td.C(0:[{\\\"dummyv1\\\": \\\"one\\\", \\\"dummyv2\\\": \\\"two\\\"}]) dependencies [B] completed" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.332Z" level=info msg="Pod node dag-diamond-jsontest-kf7td-913068289 initialized Pending" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.341Z" level=info msg="Created pod: dag-diamond-jsontest-kf7td.C(0:[{\\\"dummyv1\\\": \\\"one\\\", \\\"dummyv2\\\": \\\"two\\\"}]) (dag-diamond-jsontest-kf7td-913068289)" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.341Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.341Z" level=info msg=reconcileAgentPod namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.344Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=6782 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:21.350Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/dag-diamond-jsontest-kf7td-3988660372/labelPodCompleted
time="2022-08-30T23:35:31.343Z" level=info msg="Processing workflow" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.343Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.343Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.343Z" level=info msg="node changed" new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-diamond-jsontest-kf7td-913068289 old.message= old.phase=Pending old.progress=0/1
time="2022-08-30T23:35:31.345Z" level=info msg="node dag-diamond-jsontest-kf7td-1772556063 phase Running -> Succeeded" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.345Z" level=info msg="node dag-diamond-jsontest-kf7td-1772556063 finished: 2022-08-30 23:35:31.345323089 +0000 UTC" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.345Z" level=info msg="Skipped node dag-diamond-jsontest-kf7td-1722223206 initialized Error (message: failed to resolve {{item.dummyv2}})" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Outbound nodes of dag-diamond-jsontest-kf7td set to [dag-diamond-jsontest-kf7td-1722223206]" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="node dag-diamond-jsontest-kf7td phase Running -> Error" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="node dag-diamond-jsontest-kf7td finished: 2022-08-30 23:35:31.346203587 +0000 UTC" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Checking daemoned children of dag-diamond-jsontest-kf7td" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg=reconcileAgentPod namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Updated phase Running -> Error" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Marking workflow completed" namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.346Z" level=info msg="Checking daemoned children of " namespace=default workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.352Z" level=info msg="cleaning up pod" action=deletePod key=default/dag-diamond-jsontest-kf7td-1340600742-agent/deletePod
time="2022-08-30T23:35:31.381Z" level=info msg="Workflow update successful" namespace=default phase=Error resourceVersion=6821 workflow=dag-diamond-jsontest-kf7td
time="2022-08-30T23:35:31.393Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/dag-diamond-jsontest-kf7td-913068289/labelPodCompleted

# If the workflow's pods have not been created, you can skip the rest of the diagnostics.

# The workflow's pods that are problematic:
kubectl get pod -o yaml -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

# Logs from in your workflow's wait container, something like:
kubectl logs -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```

Here is an example i meant for how commenting out one of the withParams for task C will now permit this workflow to complete for task D. Even though it should have changed nothing for task D as far as I understand it

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: dag-diamond-jsontest-
annotations:
workflows.argoproj.io/description: |
This is an example of of breaking withParam.
spec:
serviceAccountName: workflow
entrypoint: diamond
templates:
- name: diamond
dag:
tasks:
- name: A
template: writeJson
arguments:
parameters:
- name: dummyv1
value: "dv1"
- name: dummyv2
value: "dv2"
- name: B
arguments:
parameters:
- name: dummyv1
value: "tmp"
- name: dummyv2
value: "tmp2"
- name: jsonOut
value: "{{item}}"
withParam: "{{tasks.A.outputs.parameters.stepout}}"
depends: "A"
template: writeJson
- name: C
arguments:
parameters:
- name: dummyv1
value: "tmp"
- name: dummyv2
value: "tmp2"
- name: jsonOut
value: "{{item}}"
# withParam: "{{tasks.B.outputs.parameters.stepout}}"
depends: "B"
template: writeJson
- name: D
arguments:
parameters:
- name: dummyv1
value: "{{item.dummyv1}}"
- name: dummyv2
value: "{{item.dummyv2}}"
withParam: "{{tasks.C.outputs.parameters.stepout}}"
depends: "B && C"
template: writeJson

- name: writeJson
inputs:
parameters:
- name: dummyv1
- name: dummyv2
script:
image: python:alpine3.6
command: [python]
source: |
import json
args = [dict(dummyv1='one', dummyv2='two')]
with open("/tmp/myfile.json", "w") as fd:
tmp = json.dumps(args)
fd.write(tmp)
outputs:
parameters:
- name: stepout
valueFrom:
path: /tmp/myfile.json
```
```bash
STEP TEMPLATE PODNAME DURATION MESSAGE
✔ dag-diamond-jsontest-nklct diamond
├─✔ A writeJson dag-diamond-jsontest-nklct-3272336441 4s
├─✔ B(0:dummyv1:one,dummyv2:two) writeJson dag-diamond-jsontest-nklct-2037240488 4s
├─✔ C writeJson dag-diamond-jsontest-nklct-3238781203 4s
└─✔ D(0:dummyv1:one,dummyv2:two) writeJson dag-diamond-jsontest-nklct-3224523118 4s
```

As far as I can tell reading the docs I never came across a section that seemed to imply there was a limitation on how many times you could reference using a json file as input and reading it via withParams within a single workflow

---

**Message from the maintainers**:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.