argoproj / argoproj/argo-workflows
`MetricsError: unable to compute` workflow.status in real-time metrics block with 'when' clause
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [X] I have double-checked my configuration
- [X] I can confirm the issue exists when I tested with `:latest`
- [X] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened/what did you expect to happen?
Using the `workflow.status` variable in the Prometheus metrics definition results in an error:
```
MetricsError: unable to compute 'when' clause for metric 'lovely-whale': Invalid 'when' expression ' == Running': Cannot transition token types from UNKNOWN [] to COMPARATOR [==], unable to compute 'when' clause for metric 'lovely-whale': Invalid 'when' expression ' == Running': Cannot transition token types from UNKNOWN [] to COMPARATOR [==], unable to compute 'when' clause for metric 'lovely-whale': Invalid 'when' expression ' == Running': Cannot transition token types from UNKNOWN [] to COMPARATOR [==]
```
### Version
v3.5.4
### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
```YAML
metadata:
name: lovely-whale
namespace: default
annotations:
workflows.argoproj.io/pod-name-format: v2
spec:
templates:
- name: argosay
inputs:
parameters:
- name: message
value: '{{workflow.parameters.message}}'
outputs: {}
metadata: {}
container:
name: main
image: argoproj/argosay:v2
command:
- /argosay
args:
- echo
- '{{inputs.parameters.message}}'
resources: {}
entrypoint: argosay
arguments:
parameters:
- name: message
value: hello argo
serviceAccountName: default
ttlStrategy:
secondsAfterCompletion: 300
podGC:
strategy: OnPodCompletion
metrics:
prometheus:
- name: job_duration
labels:
- key: name
value: lovely-whale
help: Duration gauge by workflow name
when: '{{workflow.status}} == Running'
gauge:
value: '{{workflow.duration}}'
realtime: true
```
### Logs from the workflow controller
```
{"Phase":"","ResourceVersion":"1396711791","level":"info","msg":"Processing workflow","namespace":"default","time":"2024-02-23T12:15:58.455Z","workflow":"lovely-whale"}
{"level":"info","msg":"Task-result reconciliation","namespace":"default","numObjs":0,"time":"2024-02-23T12:15:58.458Z","workflow":"lovely-whale"}
{"level":"error","msg":"unable to compute 'when' clause for metric 'lovely-whale': Invalid 'when' expression ' == Running': Cannot transition token types from UNKNOWN [\u003cnil\u003e] to COMPARATOR [==]","namespace":"default","time":"2024-02-23T12:15:58.459Z","workflow":"lovely-whale"}
{"level":"info","msg":"Updated phase -\u003e Running","namespace":"default","time":"2024-02-23T12:15:58.459Z","workflow":"lovely-whale"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"default","time":"2024-02-23T12:15:58.459Z","workflow":"lovely-whale"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"default","time":"2024-02-23T12:15:58.459Z","workflow":"lovely-whale"}
{"level":"info","msg":"Pod node lovely-whale initialized Pending","namespace":"default","time":"2024-02-23T12:15:58.459Z","workflow":"lovely-whale"}
{"level":"info","msg":"Created pod: lovely-whale (lovely-whale)","namespace":"default","time":"2024-02-23T12:15:58.482Z","workflow":"lovely-whale"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"default","time":"2024-02-23T12:15:58.482Z","workflow":"lovely-whale"}
{"level":"info","msg":"reconcileAgentPod","namespace":"default","time":"2024-02-23T12:15:58.482Z","workflow":"lovely-whale"}
{"level":"info","msg":"Workflow update successful","namespace":"default","phase":"Running","resourceVersion":"1396711796","time":"2024-02-23T12:15:58.492Z","workflow":"lovely-whale"}
{"Phase":"Running","ResourceVersion":"1396711796","level":"info","msg":"Processing workflow","namespace":"default","time":"2024-02-23T12:16:08.483Z","workflow":"lovely-whale"}
{"level":"info","msg":"Task-result reconciliation","namespace":"default","numObjs":1,"time":"2024-02-23T12:16:08.483Z","workflow":"lovely-whale"}
{"level":"info","msg":"task-result changed","namespace":"default","nodeID":"lovely-whale","time":"2024-02-23T12:16:08.483Z","workflow":"lovely-whale"}
{"level":"error","msg":"unable to compute 'when' clause for metric 'lovely-whale': Invalid 'when' expression ' == Running': Cannot transition token types from UNKNOWN [\u003cnil\u003e] to COMPARATOR [==]","namespace":"default","time":"2024-02-23T12:16:08.483Z","workflow":"lovely-whale"}
{"level":"info","msg":"node changed","namespace":"default","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"lovely-whale","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-02-23T12:16:08.483Z","workflow":"lovely-whale"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"default","time":"2024-02-23T12:16:08.483Z","workflow":"lovely-whale"}
{"level":"info","msg":"reconcileAgentPod","namespace":"default","time":"2024-02-23T12:16:08.483Z","workflow":"lovely-whale"}
{"level":"info","msg":"Updated phase Running -\u003e Succeeded","namespace":"default","time":"2024-02-23T12:16:08.483Z","workflow":"lovely-whale"}
{"level":"info","msg":"Marking workflow completed","namespace":"default","time":"2024-02-23T12:16:08.483Z","workflow":"lovely-whale"}
{"level":"info","msg":"Marking workflow as pending archiving","namespace":"default","time":"2024-02-23T12:16:08.483Z","workflow":"lovely-whale"}
{"level":"error","msg":"unable to compute 'when' clause for metric 'lovely-whale': Invalid 'when' expression ' == Running': Cannot transition token types from UNKNOWN [\u003cnil\u003e] to COMPARATOR [==]","namespace":"default","time":"2024-02-23T12:16:08.484Z","workflow":"lovely-whale"}
{"action":"deletePod","key":"default/lovely-whale-1340600742-agent/deletePod","level":"info","msg":"cleaning up pod","time":"2024-02-23T12:16:08.489Z"}
{"level":"info","msg":"Workflow update successful","namespace":"default","phase":"Succeeded","resourceVersion":"1396711974","time":"2024-02-23T12:16:08.493Z","workflow":"lovely-whale"}
{"level":"info","msg":"archiving workflow","namespace":"default","time":"2024-02-23T12:16:08.500Z","uid":"2bc6d6f9-2ab3-4a0e-9051-f353bdc39f7f","workflow":"lovely-whale"}
{"level":"info","msg":"Queueing Succeeded workflow default/lovely-whale for delete in 5m0s due to TTL","time":"2024-02-23T12:16:08.526Z"}
{"action":"deletePod","key":"default/lovely-whale/deletePod","level":"info","msg":"cleaning up pod","time":"2024-02-23T12:16:13.501Z"}
{"level":"info","msg":"Queueing Succeeded workflow default/lovely-whale for delete due to max rention(10 workflows)","time":"2024-02-23T12:19:20.461Z"}
{"level":"info","msg":"Deleting garbage collected workflow 'default/lovely-whale'","time":"2024-02-23T12:19:20.461Z"}
{"level":"info","msg":"Successfully request 'default/lovely-whale' to be deleted","time":"2024-02-23T12:19:20.467Z"}
{"level":"info","msg":"Deleting garbage collected workflow 'default/lovely-whale'","time":"2024-02-23T12:21:09.000Z"}
{"level":"info","msg":"Workflow already deleted 'default/lovely-whale'","time":"2024-02-23T12:21:09.004Z"}
```
Contributor guide
Assessment
This issue has not been assessed yet.