argoproj / argoproj/argo-workflows
should emit realtime metrics while node not fulfilled
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [X] I have double-checked my configuration
- [X] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [X] I have searched existing issues and could not find a match for this bug
- [X] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
**What happened?**
When I use the realtime metrics in templates level. The realtime metrics not emit in every template execution.
Here is a part of my workflow template
```yaml
spec:
templates:
- name: gen-random-int
retryStrategy:
limit: 11
retryPolicy: Always
backoff:
duration: '20'
factor: 2
maxDuration: 20m
affinity:
nodeAntiAffinity: {}
metrics:
prometheus:
- name: node_timeout_metrics
help: Duration gauge by name
when: '{{duration}} > 100'
gauge:
value: '{{duration}}'
realtime: true
```
**What did you expect to happen?**
I think the realtime metrics need to be emitted as timely as possible.
Look like above metrics configuration, i need when `{{duration}} > 100` , than emit the metrics. But Currently, it only supports sending after the node `Fulfilled`.
### Version(s)
main
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
```YAML
templates:
- name: gen-random-int
retryStrategy:
limit: 11
retryPolicy: Always
backoff:
duration: '20'
factor: 2
maxDuration: 20m
affinity:
nodeAntiAffinity: {}
metrics:
prometheus:
- name: node_timeout_metrics
help: Duration gauge by name
when: '{{duration}} > 100'
gauge:
value: '{{duration}}'
realtime: true
```
### Logs from the workflow controller
```text
null
```
### Logs from in your workflow's wait container
```text
null
```
Contributor guide
Research direction
Start by reproducing the minimal workflow with the template-level Prometheus gauge using `realtime: true`, then trace how the workflow controller evaluates metrics before and after node fulfillment. Done means the `when: '{{duration}} > 100'` metric is emitted during execution rather than only after the node is fulfilled, with a regression test covering the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, prometheus
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100