argoproj / argoproj/argo-workflows
Retrying a timeout suspended node with different user input doesn't spin up the subsequent nodes
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
When the workflow's suspend node times out due to no user input received; we skip the subsequent workflows. But if we retry the suspend node and give another input; the further nodes do not spin up if the input is changed.
Before retrying the timed out node
After retrying and supplying different input
Nothing spin up post that and there are no useful logs in either controller or containers
### Version(s)
v3.6.2
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: minimal-suspend-timeout-
spec:
entrypoint: main
templates:
- name: main
steps:
- - name: wait
template: wait
- - name: after
template: after
# Suspends for 5 seconds, then times out and fails.
- name: wait
suspend:
duration: "5s"
- name: after
container:
image: docker/whalesay
command: [cowsay]
args: ["Hello, I'm after the suspend!"]
```
### Logs from the workflow controller
```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
There are no logs emmitted for this event in workflow
```
### Logs from in your workflow's wait container
```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```
Contributor guide
Research direction
Start by running the minimal suspend-timeout workflow from the issue on Argo Workflows v3.6.2 or latest, then retry the timed-out suspend node with different input. Inspect the workflow-controller handling for suspend-node retries and compare the workflow status and controller logs; done means the subsequent `after` node starts after the changed input is supplied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100