argoproj / argoproj/argo-workflows
Daemon steps aren't restarted when retrying a workflow using `argo retry`
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [X] I have double-checked my configuration
- [X] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [X] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
I have a simple two step workflow. The first step is a daemon process, the second step always fails after some time. When the second step fails, as expected the workflow fails and the daemon is stopped. If I `argo retry` the workflow the failed second step will restart, but the daemon step will not restart. Even targeting the daemon step using `--node-field-selector` will not restart the daemon step.
Failed workflow status
```
➜ argo get -n default restart-daemon-example-x9f4n
Name: restart-daemon-example-x9f4n
Namespace: default
ServiceAccount: unset (will run with the default ServiceAccount)
Status: Failed
Message: child 'restart-daemon-example-x9f4n-1496432179' failed
Conditions:
PodRunning False
Completed True
Created: Wed Aug 14 14:37:41 -0500 (47 seconds ago)
Started: Wed Aug 14 14:37:41 -0500 (47 seconds ago)
Finished: Wed Aug 14 14:38:12 -0500 (16 seconds ago)
Duration: 31 seconds
Progress: 1/2
ResourcesDuration: 0s*(1 cpu),15s*(100Mi memory)
STEP TEMPLATE PODNAME DURATION MESSAGE
✖ restart-daemon-example-x9f4n main child 'restart-daemon-example-x9f4n-1496432179' failed
├───✔ monitor monitor restart-daemon-example-x9f4n-monitor-3740220798 10s
└───✖ fail fail restart-daemon-example-x9f4n-fail-1496432179 11s Error (exit code 1)
```
both `argo retry restart-daemon-example-x9f4n` and `argo retry restart-daemon-example-x9f4n --node-field-selector=displayName=monitor` only result in fail being rerun. the monitor daemon is not restarted.
Adding in `--restart-successful` results in the daemon node going away entirely.
Output from the retry command `argo retry restart-daemon-example-x9f4n --node-field-selector=displayName=monitor --restart-successful`
```
Name: restart-daemon-example-x9f4n
Namespace: default
ServiceAccount: unset (will run with the default ServiceAccount)
Status: Running
Conditions:
PodRunning True
Completed False
Created: Wed Aug 14 14:37:41 -0500 (17 hours ago)
Started: Thu Aug 15 08:09:48 -0500 (12 seconds ago)
Duration: 12 seconds
Progress: 0/1
STEP TEMPLATE PODNAME DURATION MESSAGE
● restart-daemon-example-x9f4n main
└─────● fail
```
The fail steps then fails, as expected.
```
➜ fund-rating-megaphone git:(main) ✗ argo get -n default restart-daemon-example-x9f4n
Name: restart-daemon-example-x9f4n
Namespace: default
ServiceAccount: unset (will run with the default ServiceAccount)
Status: Failed
Message: child 'restart-daemon-example-x9f4n-1496432179' failed
Conditions:
PodRunning False
Completed True
Created: Wed Aug 14 14:37:41 -0500 (17 hours ago)
Started: Thu Aug 15 08:09:48 -0500 (1 minute ago)
Finished: Thu Aug 15 08:10:09 -0500 (1 minute ago)
Duration: 21 seconds
Progress: 0/1
ResourcesDuration: 0s*(1 cpu),14s*(100Mi memory)
STEP TEMPLATE PODNAME DURATION MESSAGE
✖ restart-daemon-example-x9f4n main child 'restart-daemon-example-x9f4n-1496432179' failed
└─────✖ fail fail restart-daemon-example-x9f4n-fail-1496432179 11s Error (exit code 1)
```
Based on [conversation in the CNCF slack](https://cloud-native.slack.com/archives/C01QW9QSSSK/p1723664559914669) channel this behaviour is likely unexpected.
### Version(s)
v3.5.10, latest
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: restart-daemon-example-
namespace: default
spec:
entrypoint: main
templates:
- name: main
steps:
- - name: monitor
template: monitor
- - name: fail
template: fail
- name: monitor
daemon: true
script:
image: python:alpine3.6
command: [ python ]
source: |
import time
while True:
print(f"Daemon updating monitoring status", flush=True)
time.sleep(5)
- name: fail
script:
image: python:alpine3.6
command: [ python ]
source: |
import time
import sys
time.sleep(5)
sys.exit(1)
```
### Logs from the workflow controller
```text
Initial logs
time="2024-08-19T15:21:34.136Z" level=info msg="Processing workflow" Phase= ResourceVersion=969 namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:34.138Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:34.138Z" level=info msg="Updated phase -> Running" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:34.138Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:34.138Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:34.138Z" level=info msg="Steps node restart-daemon-example-5bb5b initialized Running" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:34.138Z" level=info msg="StepGroup node restart-daemon-example-5bb5b-2839827893 initialized Running" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:34.138Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:34.138Z" level=info msg="Pod node restart-daemon-example-5bb5b-4135233141 initialized Pending" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:34.143Z" level=info msg="Created pod: restart-daemon-example-5bb5b[0].monitor (restart-daemon-example-5bb5b-monitor-4135233141)" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:34.143Z" level=info msg="Workflow step group node restart-daemon-example-5bb5b-2839827893 not yet completed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:34.143Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:34.143Z" level=info msg=reconcileAgentPod namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:34.146Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=973 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:44.138Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=973 namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:44.138Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:44.139Z" level=info msg="node changed" namespace=default new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=restart-daemon-example-5bb5b-4135233141 old.message= old.phase=Pending old.progress=0/1 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:44.139Z" level=info msg="Workflow step group node restart-daemon-example-5bb5b-2839827893 not yet completed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:44.139Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:44.139Z" level=info msg=reconcileAgentPod namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:44.148Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=994 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.152Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=994 namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.153Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.153Z" level=info msg="Node became daemoned" namespace=default nodeId=restart-daemon-example-5bb5b-4135233141 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.154Z" level=info msg="node changed" namespace=default new.message= new.phase=Running new.progress=0/1 nodeID=restart-daemon-example-5bb5b-4135233141 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.155Z" level=info msg="Step group node restart-daemon-example-5bb5b-2839827893 successful" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.155Z" level=info msg="node restart-daemon-example-5bb5b-2839827893 phase Running -> Succeeded" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.156Z" level=info msg="node restart-daemon-example-5bb5b-2839827893 finished: 2024-08-19 15:21:54.1559365 +0000 UTC" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.156Z" level=info msg="StepGroup node restart-daemon-example-5bb5b-2772864512 initialized Running" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.156Z" level=info msg="SG Outbound nodes of restart-daemon-example-5bb5b-4135233141 are [restart-daemon-example-5bb5b-4135233141]" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.156Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.157Z" level=info msg="Pod node restart-daemon-example-5bb5b-1475243306 initialized Pending" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.166Z" level=info msg="Created pod: restart-daemon-example-5bb5b[1].fail (restart-daemon-example-5bb5b-fail-1475243306)" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.166Z" level=info msg="Workflow step group node restart-daemon-example-5bb5b-2772864512 not yet completed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.166Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.166Z" level=info msg=reconcileAgentPod namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:21:54.172Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=1006 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:04.169Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1006 namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:04.169Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:04.170Z" level=info msg="node changed" namespace=default new.message= new.phase=Running new.progress=0/1 nodeID=restart-daemon-example-5bb5b-1475243306 old.message= old.phase=Pending old.progress=0/1 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:04.170Z" level=info msg="node unchanged" namespace=default nodeID=restart-daemon-example-5bb5b-4135233141 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:04.170Z" level=info msg="SG Outbound nodes of restart-daemon-example-5bb5b-4135233141 are [restart-daemon-example-5bb5b-4135233141]" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:04.171Z" level=info msg="Workflow step group node restart-daemon-example-5bb5b-2772864512 not yet completed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:04.171Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:04.171Z" level=info msg=reconcileAgentPod namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:04.176Z" level=info msg="cleaning up pod" action=terminateContainers key=default/restart-daemon-example-5bb5b-fail-1475243306/terminateContainers
time="2024-08-19T15:22:04.180Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=1033 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.852Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1033 namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.852Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.852Z" level=info msg="Pod failed: Error (exit code 1)" displayName=fail namespace=default pod=restart-daemon-example-5bb5b-fail-1475243306 templateName=fail workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.852Z" level=info msg="node changed" namespace=default new.message="Error (exit code 1)" new.phase=Failed new.progress=0/1 nodeID=restart-daemon-example-5bb5b-1475243306 old.message= old.phase=Running old.progress=0/1 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.852Z" level=info msg="node unchanged" namespace=default nodeID=restart-daemon-example-5bb5b-4135233141 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.852Z" level=info msg="SG Outbound nodes of restart-daemon-example-5bb5b-4135233141 are [restart-daemon-example-5bb5b-4135233141]" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.852Z" level=info msg="Step group node restart-daemon-example-5bb5b-2772864512 deemed failed: child 'restart-daemon-example-5bb5b-1475243306' failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.852Z" level=info msg="node restart-daemon-example-5bb5b-2772864512 phase Running -> Failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.852Z" level=info msg="node restart-daemon-example-5bb5b-2772864512 message: child 'restart-daemon-example-5bb5b-1475243306' failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.852Z" level=info msg="node restart-daemon-example-5bb5b-2772864512 finished: 2024-08-19 15:22:14.852953427 +0000 UTC" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.852Z" level=info msg="step group restart-daemon-example-5bb5b-2772864512 was unsuccessful: child 'restart-daemon-example-5bb5b-1475243306' failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.852Z" level=info msg="Outbound nodes of restart-daemon-example-5bb5b-1475243306 is [restart-daemon-example-5bb5b-1475243306]" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.852Z" level=info msg="Outbound nodes of restart-daemon-example-5bb5b is [restart-daemon-example-5bb5b-1475243306]" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.852Z" level=info msg="node restart-daemon-example-5bb5b phase Running -> Failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.853Z" level=info msg="node restart-daemon-example-5bb5b message: child 'restart-daemon-example-5bb5b-1475243306' failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.853Z" level=info msg="node restart-daemon-example-5bb5b finished: 2024-08-19 15:22:14.853009552 +0000 UTC" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.853Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.853Z" level=info msg=reconcileAgentPod namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.853Z" level=info msg="Updated phase Running -> Failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.853Z" level=info msg="Updated message -> child 'restart-daemon-example-5bb5b-1475243306' failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.853Z" level=info msg="Marking workflow completed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.858Z" level=info msg="cleaning up pod" action=terminateContainers key=default/restart-daemon-example-5bb5b-monitor-4135233141/terminateContainers
time="2024-08-19T15:22:14.859Z" level=info msg="Workflow update successful" namespace=default phase=Failed resourceVersion=1043 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:22:14.859Z" level=info msg="https://10.43.0.1:443/api/v1/namespaces/default/pods/restart-daemon-example-5bb5b-monitor-4135233141/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec&command=kill&command=15&command=1&container=main&stderr=true&stdout=true&tty=false"
time="2024-08-19T15:22:14.870Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/restart-daemon-example-5bb5b-monitor-4135233141/labelPodCompleted
time="2024-08-19T15:22:14.870Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/restart-daemon-example-5bb5b-fail-1475243306/labelPodCompleted
time="2024-08-19T15:22:14.909Z" level=info msg="signaled container" container=main error="" namespace=default pod=restart-daemon-example-5bb5b-monitor-4135233141 stderr= stdout="killing 1 with terminated\n"
time="2024-08-19T15:22:14.909Z" level=info msg="https://10.43.0.1:443/api/v1/namespaces/default/pods/restart-daemon-example-5bb5b-monitor-4135233141/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec&command=kill&command=15&command=1&container=wait&stderr=true&stdout=true&tty=false"
time="2024-08-19T15:22:14.951Z" level=info msg="signaled container" container=wait error="" namespace=default pod=restart-daemon-example-5bb5b-monitor-4135233141 stderr= stdout="killing 1 with terminated\n"
time="2024-08-19T15:22:34.177Z" level=info msg="cleaning up pod" action=killContainers key=default/restart-daemon-example-5bb5b-fail-1475243306/killContainers
```
Logs after retry
```
time="2024-08-19T15:24:56.223Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1137 namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:24:56.223Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:24:56.223Z" level=info msg="Step 'nil' has no expanded child nodes" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:24:56.223Z" level=info msg="StepGroup node restart-daemon-example-5bb5b-2772864512 initialized Running" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:24:56.223Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:24:56.223Z" level=info msg="Pod node restart-daemon-example-5bb5b-1475243306 initialized Pending" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:24:56.229Z" level=info msg="Created pod: restart-daemon-example-5bb5b[1].fail (restart-daemon-example-5bb5b-fail-1475243306)" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:24:56.229Z" level=info msg="Workflow step group node restart-daemon-example-5bb5b-2772864512 not yet completed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:24:56.229Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:24:56.229Z" level=info msg=reconcileAgentPod namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:24:56.233Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=1140 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:06.224Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1140 namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:06.225Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:06.226Z" level=info msg="node changed" namespace=default new.message= new.phase=Running new.progress=0/1 nodeID=restart-daemon-example-5bb5b-1475243306 old.message= old.phase=Pending old.progress=0/1 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:06.226Z" level=info msg="Step 'nil' has no expanded child nodes" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:06.226Z" level=info msg="SG Outbound nodes of restart-daemon-example-5bb5b-2772864512 are []" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:06.227Z" level=info msg="Workflow step group node restart-daemon-example-5bb5b-2772864512 not yet completed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:06.227Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:06.227Z" level=info msg=reconcileAgentPod namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:06.232Z" level=info msg="cleaning up pod" action=terminateContainers key=default/restart-daemon-example-5bb5b-fail-1475243306/terminateContainers
time="2024-08-19T15:25:06.233Z" level=info msg="https://10.43.0.1:443/api/v1/namespaces/default/pods/restart-daemon-example-5bb5b-fail-1475243306/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec&command=kill&command=15&command=1&container=wait&stderr=true&stdout=true&tty=false"
time="2024-08-19T15:25:06.236Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=1164 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:06.243Z" level=info msg="signaled container" container=wait error="unable to upgrade connection: container not found (\"wait\")" namespace=default pod=restart-daemon-example-5bb5b-fail-1475243306 stderr="" stdout=""
time="2024-08-19T15:25:16.425Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1164 namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.425Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.426Z" level=info msg="Pod failed: Error (exit code 1)" displayName=fail namespace=default pod=restart-daemon-example-5bb5b-fail-1475243306 templateName=fail workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.426Z" level=info msg="node changed" namespace=default new.message="Error (exit code 1)" new.phase=Failed new.progress=0/1 nodeID=restart-daemon-example-5bb5b-1475243306 old.message= old.phase=Running old.progress=0/1 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.426Z" level=info msg="Step 'nil' has no expanded child nodes" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.426Z" level=info msg="SG Outbound nodes of restart-daemon-example-5bb5b-2772864512 are []" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg="Step group node restart-daemon-example-5bb5b-2772864512 deemed failed: child 'restart-daemon-example-5bb5b-1475243306' failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg="node restart-daemon-example-5bb5b-2772864512 phase Running -> Failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg="node restart-daemon-example-5bb5b-2772864512 message: child 'restart-daemon-example-5bb5b-1475243306' failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg="node restart-daemon-example-5bb5b-2772864512 finished: 2024-08-19 15:25:16.427367136 +0000 UTC" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg="step group restart-daemon-example-5bb5b-2772864512 was unsuccessful: child 'restart-daemon-example-5bb5b-1475243306' failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg="Outbound nodes of restart-daemon-example-5bb5b-1475243306 is [restart-daemon-example-5bb5b-1475243306]" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg="Outbound nodes of restart-daemon-example-5bb5b is [restart-daemon-example-5bb5b-1475243306]" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg="node restart-daemon-example-5bb5b phase Running -> Failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg="node restart-daemon-example-5bb5b message: child 'restart-daemon-example-5bb5b-1475243306' failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg="node restart-daemon-example-5bb5b finished: 2024-08-19 15:25:16.427473219 +0000 UTC" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg=reconcileAgentPod namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg="Updated phase Running -> Failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg="Updated message -> child 'restart-daemon-example-5bb5b-1475243306' failed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.427Z" level=info msg="Marking workflow completed" namespace=default workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.438Z" level=info msg="Workflow update successful" namespace=default phase=Failed resourceVersion=1175 workflow=restart-daemon-example-5bb5b
time="2024-08-19T15:25:16.448Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/restart-daemon-example-5bb5b-fail-1475243306/labelPodCompleted
```
```
### Logs from in your workflow's wait container
argo-server-6565db57c9-n9f7x
```text
➜ ~ kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=restart-daemon-example-5bb5b,workflow.argoproj.io/phase!=Succeeded
No resources found in argo namespace.
```
Logs after retry
```
➜ ~ kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=restart-daemon-example-5bb5b,workflow.argoproj.io/phase!=Succeeded
No resources found in argo namespace.
```
Contributor guide
Assessment
This issue has not been assessed yet.