argoproj / argoproj/argo-workflows

once-off error under load, `{{retries}}` not replaced

Open
#13,799 3 comments 0 reactions 0 assignees View on GitHub
area/retryStrategy area/templating solution/outdated
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [X] I have double-checked my configuration
- [ ] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [X] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

my wf is a dag and has snippet like:

```yaml
templates:
- name: redact-wf
dag:
tasks:
- name: redact
depends: redact.Succeeded
templateRef:
name: redact
template: main
arguments:
parameters:
- name: redact_name
value: "redact-{{workflow.name}}-{{retries}}"
```

There have been 100s of successful runs of this workflow but only 1 run where it failed (my own code inside tried to parse the section after last hyphen as int) as the parameter that went into the pod did not have the `{{retries}}` value substituted, strangely the ui shows it as replaced with the 0 for retries but pod logs below show that it wasn't

controller logs:

```
time=\"2024-10-21T20:01:58.682Z\" level=info msg=\"Transient error: Operation cannot be fulfilled on resourcequotas \\\"myresourcequota\\\": the object has been modified; please apply your changes to the latest version and try again\"
time=\"2024-10-21T20:01:58.682Z\" level=info msg=\"Transient error: Operation cannot be fulfilled on resourcequotas \\\"myresourcequota\\\": the object has been modified; please apply your changes to the latest version and try again\"
time=\"2024-10-21T20:01:58.682Z\" level=info msg=\"Mark node redact(0).redact(0)[1].redact(0) as Pending, due to: Operation cannot be fulfilled on resourcequotas \\\"myresourcequota\\\": the object has been modified; please apply your changes to the latest version and try again\" namespace=redact workflow=redact
time=\"2024-10-21T20:01:58.683Z\" level=info msg=\"node redact-redactid message: Operation cannot be fulfilled on resourcequotas \\\"myresourcequota\\\": the object has been modified; please apply your changes to the latest version and try again\" namespace=redact workflow=redact
time=\"2024-10-21T20:02:17.877Z\" level=info msg=\"node changed\" namespace=redact new.message= new.phase=Running new.progress=0/1 nodeID=redact-redactid old.message=\"Operation cannot be fulfilled on resourcequotas \\\"myresourcequota\\\": the object has been modified; please apply your changes to the latest version and try again\" old.phase=Pending old.progress=0/1 workflow=redact
```

my pod logs show that the `{{retries}}` was not properly replaced with 0:

`time=\"2024-10-21T20:02:08.957Z\" level=info msg=\"Executor initialized\" deadline=\"2024-10-redact 09:00:45 +0000 UTC\" includeScriptOutput=false namespace=redact podName=redact template=\"{\\\"name\\\":\\\"redact\\\",\\\"inputs\\\":{\\\"parameters\\\":[{\\\"name\\\":\\\"job_name\\\",\\\"value\\\":\\\"redactwf-redact-{{retries}}\\\"}`

my resourcequota limits were not hit but the controller was busy with 100s of "cleaning up pod" from a different workflow

may be related? https://github.com/argoproj/argo-workflows/blob/v3.4.11/util/template/expression_template.go#L35-L40 seems like allowUnresolved is passed in as true at https://github.com/argoproj/argo-workflows/blame/v3.4.11/workflow/common/util.go#L286
https://github.com/argoproj/argo-workflows/issues/13123 but i don't use podspecpatch

it was able to replace `{{workflow.name}}` but not `{{retries}}`

### Version(s)

3.4.11

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
n/a
```

### Logs from the workflow controller

```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
n/a
```

### Logs from in your workflow's wait container

```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
n/a
```

Contributor guide

Open the contributing guide

Research direction

Start with util/template/expression_template.go and workflow/common/util.go, especially the allowUnresolved path mentioned in the report, then compare template handling during the resource-quota conflict shown in the controller logs. Reproduce the DAG parameter case if possible; done means determining why {{retries}} remains unresolved while workflow.name is substituted and verifying the behavior is corrected.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.