argoproj / argoproj/argo-workflows
Manual retry logic is abnormal when `Steps` or `DAG` are configured with `RetryStrategy`
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [x] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
As shown in the example, `workflow-steps-with-retry-fail` sets a retryStrategy on `Steps` `fail-step-group`.
After execution, it will probably end up failing, as shown in the figure below.
After manual retry, it transitions to the following state.
After #13738, `childrenFulfilled` checks if all the nodes children are fulfilled, which will cause problems during manual retries, similar to #14796.
https://github.com/argoproj/argo-workflows/blob/41e1e8eea386c792a4f5d9c51664da6b78f90e60/workflow/controller/operator.go#L2206
### Version(s)
41e1e8e
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: workflow-steps-with-retry-fail
spec:
entrypoint: main
templates:
- name: main
steps:
- - name: retry-step-group-case
template: fail-step-group
- name: fail-with-rate
container:
image: python:alpine3.6
command: ["python", -c]
args: ["import random; import sys; exit_code = random.choice([0, 1]); sys.exit(exit_code);"]
- name: fail-step-group
steps:
- - name: step1
template: fail-with-rate
- - name: step2
template: fail-with-rate
- - name: step3
template: fail-with-rate
retryStrategy:
limit: "1"
```
### Logs from the workflow controller
```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
```
### Logs from in your workflow's wait container
```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```
Contributor guide
Research direction
Start at workflow/controller/operator.go around line 2206, where childrenFulfilled is checked during manual retry, and reproduce the behavior with the provided workflow-steps-with-retry-fail manifest. Compare the workflow state before and after manual retry; done means retrying a failed Steps or DAG with RetryStrategy reaches the expected state without the abnormal transition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100