argoproj / argoproj/argo-workflows
retryStrategy does not work if a workflow level lifecycle hook is present
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [X] I have double-checked my configuration
- [X] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [X] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened/what did you expect to happen?
If there is a retryStrategy in the templateDefaults, then tasks should retry, even if there is also a workflow level lifecycle hook.
In the example workflow, if I remove the hook, then it behaves as expected, retrying 1 time.
Strangely, if I move the retryStrategy out of templateDefaults and put it in directly in the "main" template, then it does work, even with the lifecycle hook present.
But I cannot get it to work with the combination I've shown, with a retryStrategy in the templateDefaults along with a workflow level lifecycle hook.
### Version
latest
### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: reproduce-bug-
spec:
entrypoint: main
hooks:
running:
expression: workflow.status == "Running"
template: notify
templateDefaults:
retryStrategy:
limit: 1
templates:
- name: notify
script:
image: alpine
command: [sh]
source: |
echo "notifying workflow is {{workflow.status}}..."
- name: main
script:
image: alpine
command: [sh]
source: |
sleep 5
echo "raising an error..."
exit 1
```
### Logs from the workflow controller
```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
time="2024-06-21T20:59:44.653Z" level=info msg="Processing workflow" Phase= ResourceVersion=2952083 namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.655Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.655Z" level=info msg="Updated phase -> Running" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.656Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.656Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.656Z" level=info msg="Retry node reproduce-bug-hn74v initialized Running" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.656Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.656Z" level=info msg="Pod node reproduce-bug-hn74v-3447748310 initialized Pending" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.656Z" level=warning msg="couldn't get boundaryTemplate through nodeName reproduce-bug-hn74v(0)" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.664Z" level=info msg="Created pod: reproduce-bug-hn74v(0) (reproduce-bug-hn74v-main-3447748310)" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.665Z" level=info msg="Running workflow level hooks" lifeCycleHook=running namespace=argo node=reproduce-bug-hn74v.hooks.running workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.665Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.665Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.665Z" level=info msg="Retry node reproduce-bug-hn74v-1571837284 initialized Running" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.665Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.665Z" level=info msg="Pod node reproduce-bug-hn74v-1097678159 initialized Pending" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.665Z" level=warning msg="couldn't get boundaryTemplate through nodeName reproduce-bug-hn74v.hooks.running(0)" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.674Z" level=info msg="Created pod: reproduce-bug-hn74v.hooks.running(0) (reproduce-bug-hn74v-notify-1097678159)" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.674Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.674Z" level=info msg=reconcileAgentPod namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:44.688Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=2952090 workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:54.656Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2952090 namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:54.656Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:54.656Z" level=info msg="task-result changed" namespace=argo nodeID=reproduce-bug-hn74v-1097678159 workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:54.656Z" level=info msg="task-result changed" namespace=argo nodeID=reproduce-bug-hn74v-3447748310 workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:54.656Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=reproduce-bug-hn74v-1097678159 old.message= old.phase=Pending old.progress=0/1 workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:54.656Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=reproduce-bug-hn74v-3447748310 old.message= old.phase=Pending old.progress=0/1 workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:54.657Z" level=info msg="Running workflow level hooks" lifeCycleHook=running namespace=argo node=reproduce-bug-hn74v.hooks.running workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:54.657Z" level=info msg="node reproduce-bug-hn74v-1571837284 phase Running -> Succeeded" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:54.657Z" level=info msg="node reproduce-bug-hn74v-1571837284 finished: 2024-06-21 20:59:54.657274293 +0000 UTC" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:54.657Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:54.657Z" level=info msg=reconcileAgentPod namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:54.667Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=2952142 workflow=reproduce-bug-hn74v
time="2024-06-21T20:59:54.674Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/reproduce-bug-hn74v-notify-1097678159/labelPodCompleted
time="2024-06-21T21:00:04.697Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2952142 namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.697Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.697Z" level=info msg="task-result changed" namespace=argo nodeID=reproduce-bug-hn74v-1097678159 workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.697Z" level=info msg="task-result changed" namespace=argo nodeID=reproduce-bug-hn74v-3447748310 workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.697Z" level=info msg="Pod failed: Error (exit code 1)" displayName="reproduce-bug-hn74v(0)" namespace=argo pod=reproduce-bug-hn74v-main-3447748310 templateName=main workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.697Z" level=info msg="node changed" namespace=argo new.message="Error (exit code 1)" new.phase=Failed new.progress=0/1 nodeID=reproduce-bug-hn74v-3447748310 old.message= old.phase=Running old.progress=0/1 workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.698Z" level=info msg="node reproduce-bug-hn74v phase Running -> Succeeded" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.698Z" level=info msg="node reproduce-bug-hn74v finished: 2024-06-21 21:00:04.698254425 +0000 UTC" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.698Z" level=info msg="Running workflow level hooks" lifeCycleHook=running namespace=argo node=reproduce-bug-hn74v.hooks.running workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.698Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.698Z" level=info msg=reconcileAgentPod namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.698Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.698Z" level=info msg="Marking workflow completed" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.698Z" level=info msg="Marking workflow as pending archiving" namespace=argo workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.704Z" level=info msg="cleaning up pod" action=deletePod key=argo/reproduce-bug-hn74v-1340600742-agent/deletePod
time="2024-06-21T21:00:04.706Z" level=warning msg="failed to clean-up pod" action=deletePod error="pods \"reproduce-bug-hn74v-1340600742-agent\" not found" key=argo/reproduce-bug-hn74v-1340600742-agent/deletePod
time="2024-06-21T21:00:04.706Z" level=warning msg="Non-transient error: pods \"reproduce-bug-hn74v-1340600742-agent\" not found"
time="2024-06-21T21:00:04.708Z" level=info msg="Workflow update successful" namespace=argo phase=Succeeded resourceVersion=2952165 workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.723Z" level=info msg="archiving workflow" namespace=argo uid=f3c36574-685b-42bf-9dc8-ad0f54eafb27 workflow=reproduce-bug-hn74v
time="2024-06-21T21:00:04.728Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/reproduce-bug-hn74v-main-3447748310/labelPodCompleted
```
### Logs from in your workflow's wait container
```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
time="2024-06-21T20:59:56.465Z" level=info msg="No output parameters"
time="2024-06-21T20:59:56.465Z" level=info msg="No output artifacts"
time="2024-06-21T20:59:56.466Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: reproduce-bug-hn74v/reproduce-bug-hn74v-main-3447748310/main.log"
time="2024-06-21T20:59:56.466Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2024-06-21T20:59:56.466Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=reproduce-bug-hn74v/reproduce-bug-hn74v-main-3447748310/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-06-21T20:59:56.474Z" level=info msg="Save artifact" artifactName=main-logs duration=8.091996ms error="" key=reproduce-bug-hn74v/reproduce-bug-hn74v-main-3447748310/main.log
time="2024-06-21T20:59:56.474Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-06-21T20:59:56.474Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-06-21T20:59:56.484Z" level=info msg="Alloc=7311 TotalAlloc=12775 Sys=20325 NumGC=4 Goroutines=10"
time="2024-06-21T20:59:56.490Z" level=info msg="Deadline monitor stopped"
time="2024-06-21T20:59:51.651Z" level=info msg="No output artifacts"
time="2024-06-21T20:59:51.651Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: reproduce-bug-hn74v/reproduce-bug-hn74v-notify-1097678159/main.log"
time="2024-06-21T20:59:51.651Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2024-06-21T20:59:51.651Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=reproduce-bug-hn74v/reproduce-bug-hn74v-notify-1097678159/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-06-21T20:59:51.658Z" level=info msg="Save artifact" artifactName=main-logs duration=6.896633ms error="" key=reproduce-bug-hn74v/reproduce-bug-hn74v-notify-1097678159/main.log
time="2024-06-21T20:59:51.658Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-06-21T20:59:51.658Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-06-21T20:59:51.668Z" level=info msg="Alloc=7591 TotalAlloc=12804 Sys=20325 NumGC=4 Goroutines=10"
time="2024-06-21T20:59:51.674Z" level=info msg="Deadline monitor stopped"
time="2024-06-21T20:59:51.674Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
```
Contributor guide
Research direction
Start in the workflow-controller reconciliation path shown in the logs, focusing on how templateDefaults.retryStrategy is applied when a workflow-level lifecycle hook runs. Reproduce the YAML workflow with the hook and default retry limit, then verify that the failing main task retries once instead of completing the workflow immediately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100