argoproj / argoproj/argo-workflows
Conditional using global parameter that changes causes workflow to freeze
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
## Summary
Consider the following workflow:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: changing-conditional-
spec:
entrypoint: plan
templates:
- name: plan
steps:
- - name: set-global-parameter
template: global-parameter-setting
arguments:
parameters:
- name: setting
value: true
- - name: change-global-parameter-conditionally
template: nested-global-parameter-setting
when: "{{workflow.outputs.parameters.parameter}}"
arguments:
parameters:
- name: setting
value: false
- name: nested-global-parameter-setting
inputs:
parameters:
- name: setting
steps:
- - name: set-global-parameter
template: global-parameter-setting
arguments:
parameters:
- name: setting
value: "{{inputs.parameters.setting}}"
- name: global-parameter-setting
inputs:
parameters:
- name: setting
container:
image: alpine:3.12
command: [sh, -c]
args: ["echo '{{inputs.parameters.setting}}' > /parameter.txt"]
outputs:
parameters:
- name: global-parameter
valueFrom:
path: /parameter.txt
globalName: parameter
```
This workflow sets a global parameter to `true` in its first step, which is then used as a condition to run the second nested step. If the second step runs, it changes the value of the global parameter to `false`.
However, when this workflow runs, it freezes on the second step, despite having completed its substep. Here is the `argo get` after it froze for about 30 seconds:
```
Name: changing-conditional-rsq2f
Namespace: virtual
ServiceAccount: default
Status: Running
Created: Tue Jan 12 16:03:11 -0500 (41 seconds ago)
Started: Tue Jan 12 16:03:11 -0500 (41 seconds ago)
Duration: 41 seconds
Progress: 2/2
ResourcesDuration: 3s*(1 cpu),3s*(100Mi memory)
Output Parameters:
parameter: false
STEP TEMPLATE PODNAME DURATION MESSAGE
● changing-conditional-rsq2f plan
├───✔ set-global-parameter global-parameter-setting changing-conditional-rsq2f-1345312722 2s
└───● change-global-parameter-conditionally nested-global-parameter-setting
└───✔ set-global-parameter global-parameter-setting changing-conditional-rsq2f-1327142919 3s
```
In addition, this is the sort of freezing that prevents `argo terminate` from changing anything -- the workflows still show up on `argo list -n virtual --running`.
Now for the variations.
- If we change the second setting from `false` to `true` (so it wouldn't actually change the global parameter), this completes successfully, which suggests that this is related to the conditional no longer being satisfied with an updated reference.
- If we make the second step *not* nested, then it also works correctly, so this seems to be a particular problem with the composite template here.
- If we replace the single-step `steps` template here with a single-task `dag` template, the same error happens.
- If we make the first template nested instead, we encounter a different bug: The workflow now complains that it can't resolve `{{workflow.outputs.parameters.parameter}}`, as if it's unable to see that the nested template can also set the global parameter. I'd be happy to spin this out as another issue if it isn't related.
## Diagnostics
I'm running the just-released Argo v2.12.4. I also backtested and this freezing bug also occurs in v2.12.3, v2.12.2, v2.11.8, v2.10.2, v2.9.5, and v2.8.2. (In some of these older versions, you need to replace the boolean values with strings.)
---
**Message from the maintainers**:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
Contributor guide
Research direction
Start with the provided workflow definition and reproduce it on the affected Argo Workflows versions using the nested conditional and changing global parameter. Inspect the workflow state with `argo get` and confirm that the workflow completes normally and can be terminated instead of remaining Running.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100