argoproj / argoproj/argo-workflows

Manual retry logic is abnormal when `Steps` or `DAG` are configured with `RetryStrategy`

Open
#14,804 1 comment 0 reactions 0 assignees View on GitHub
area/retry-manual area/retryStrategy
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [x] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

As shown in the example, `workflow-steps-with-retry-fail` sets a retryStrategy on `Steps` `fail-step-group`.
After execution, it will probably end up failing, as shown in the figure below.

Image

After manual retry, it transitions to the following state.

Image

After #13738, `childrenFulfilled` checks if all the nodes children are fulfilled, which will cause problems during manual retries, similar to #14796.

https://github.com/argoproj/argo-workflows/blob/41e1e8eea386c792a4f5d9c51664da6b78f90e60/workflow/controller/operator.go#L2206

Image

### Version(s)

41e1e8e

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: workflow-steps-with-retry-fail
spec:
entrypoint: main
templates:
- name: main
steps:
- - name: retry-step-group-case
template: fail-step-group
- name: fail-with-rate
container:
image: python:alpine3.6
command: ["python", -c]
args: ["import random; import sys; exit_code = random.choice([0, 1]); sys.exit(exit_code);"]
- name: fail-step-group
steps:
- - name: step1
template: fail-with-rate
- - name: step2
template: fail-with-rate
- - name: step3
template: fail-with-rate
retryStrategy:
limit: "1"
```

### Logs from the workflow controller

```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
```

### Logs from in your workflow's wait container

```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```

Contributor guide

Open the contributing guide

Research direction

Start at workflow/controller/operator.go around line 2206, where childrenFulfilled is checked during manual retry, and reproduce the behavior with the provided workflow-steps-with-retry-fail manifest. Compare the workflow state before and after manual retry; done means retrying a failed Steps or DAG with RetryStrategy reaches the expected state without the abnormal transition.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.