argoproj / argoproj/argo-workflows
once-off error under load, `{{retries}}` not replaced
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [X] I have double-checked my configuration
- [ ] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [X] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
my wf is a dag and has snippet like:
```yaml
templates:
- name: redact-wf
dag:
tasks:
- name: redact
depends: redact.Succeeded
templateRef:
name: redact
template: main
arguments:
parameters:
- name: redact_name
value: "redact-{{workflow.name}}-{{retries}}"
```
There have been 100s of successful runs of this workflow but only 1 run where it failed (my own code inside tried to parse the section after last hyphen as int) as the parameter that went into the pod did not have the `{{retries}}` value substituted, strangely the ui shows it as replaced with the 0 for retries but pod logs below show that it wasn't
controller logs:
```
time=\"2024-10-21T20:01:58.682Z\" level=info msg=\"Transient error: Operation cannot be fulfilled on resourcequotas \\\"myresourcequota\\\": the object has been modified; please apply your changes to the latest version and try again\"
time=\"2024-10-21T20:01:58.682Z\" level=info msg=\"Transient error: Operation cannot be fulfilled on resourcequotas \\\"myresourcequota\\\": the object has been modified; please apply your changes to the latest version and try again\"
time=\"2024-10-21T20:01:58.682Z\" level=info msg=\"Mark node redact(0).redact(0)[1].redact(0) as Pending, due to: Operation cannot be fulfilled on resourcequotas \\\"myresourcequota\\\": the object has been modified; please apply your changes to the latest version and try again\" namespace=redact workflow=redact
time=\"2024-10-21T20:01:58.683Z\" level=info msg=\"node redact-redactid message: Operation cannot be fulfilled on resourcequotas \\\"myresourcequota\\\": the object has been modified; please apply your changes to the latest version and try again\" namespace=redact workflow=redact
time=\"2024-10-21T20:02:17.877Z\" level=info msg=\"node changed\" namespace=redact new.message= new.phase=Running new.progress=0/1 nodeID=redact-redactid old.message=\"Operation cannot be fulfilled on resourcequotas \\\"myresourcequota\\\": the object has been modified; please apply your changes to the latest version and try again\" old.phase=Pending old.progress=0/1 workflow=redact
```
my pod logs show that the `{{retries}}` was not properly replaced with 0:
`time=\"2024-10-21T20:02:08.957Z\" level=info msg=\"Executor initialized\" deadline=\"2024-10-redact 09:00:45 +0000 UTC\" includeScriptOutput=false namespace=redact podName=redact template=\"{\\\"name\\\":\\\"redact\\\",\\\"inputs\\\":{\\\"parameters\\\":[{\\\"name\\\":\\\"job_name\\\",\\\"value\\\":\\\"redactwf-redact-{{retries}}\\\"}`
my resourcequota limits were not hit but the controller was busy with 100s of "cleaning up pod" from a different workflow
may be related? https://github.com/argoproj/argo-workflows/blob/v3.4.11/util/template/expression_template.go#L35-L40 seems like allowUnresolved is passed in as true at https://github.com/argoproj/argo-workflows/blame/v3.4.11/workflow/common/util.go#L286
https://github.com/argoproj/argo-workflows/issues/13123 but i don't use podspecpatch
it was able to replace `{{workflow.name}}` but not `{{retries}}`
### Version(s)
3.4.11
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
```YAML
n/a
```
### Logs from the workflow controller
```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
n/a
```
### Logs from in your workflow's wait container
```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
n/a
```
Contributor guide
Research direction
Start with util/template/expression_template.go and workflow/common/util.go, especially the allowUnresolved path mentioned in the report, then compare template handling during the resource-quota conflict shown in the controller logs. Reproduce the DAG parameter case if possible; done means determining why {{retries}} remains unresolved while workflow.name is substituted and verifying the behavior is corrected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100