argoproj / argoproj/argo-workflows

Retrying a timeout suspended node with different user input doesn't spin up the subsequent nodes

Open
#14,282 0 comments 1 reaction 0 assignees View on GitHub
area/suspend-resume
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

When the workflow's suspend node times out due to no user input received; we skip the subsequent workflows. But if we retry the suspend node and give another input; the further nodes do not spin up if the input is changed.

Before retrying the timed out node
Image

After retrying and supplying different input

Image
Nothing spin up post that and there are no useful logs in either controller or containers

### Version(s)

v3.6.2

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: minimal-suspend-timeout-
spec:
entrypoint: main
templates:
- name: main
steps:
- - name: wait
template: wait
- - name: after
template: after

# Suspends for 5 seconds, then times out and fails.
- name: wait
suspend:
duration: "5s"

- name: after
container:
image: docker/whalesay
command: [cowsay]
args: ["Hello, I'm after the suspend!"]
```

### Logs from the workflow controller

```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

There are no logs emmitted for this event in workflow
```

### Logs from in your workflow's wait container

```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```

Contributor guide

Open the contributing guide

Research direction

Start by running the minimal suspend-timeout workflow from the issue on Argo Workflows v3.6.2 or latest, then retry the timed-out suspend node with different input. Inspect the workflow-controller handling for suspend-node retries and compare the workflow status and controller logs; done means the subsequent `after` node starts after the changed input is supplied.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.