argoproj / argoproj/argo-workflows

should emit realtime metrics while node not fulfilled

Open
#13,440 5 comments 0 reactions 0 assignees View on GitHub
area/telemetry/metrics
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [X] I have searched existing issues and could not find a match for this bug
- [X] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

**What happened?**
When I use the realtime metrics in templates level. The realtime metrics not emit in every template execution.

Here is a part of my workflow template

```yaml
spec:
templates:
- name: gen-random-int
retryStrategy:
limit: 11
retryPolicy: Always
backoff:
duration: '20'
factor: 2
maxDuration: 20m
affinity:
nodeAntiAffinity: {}
metrics:
prometheus:
- name: node_timeout_metrics
help: Duration gauge by name
when: '{{duration}} > 100'
gauge:
value: '{{duration}}'
realtime: true
```

**What did you expect to happen?**

I think the realtime metrics need to be emitted as timely as possible.
Look like above metrics configuration, i need when `{{duration}} > 100` , than emit the metrics. But Currently, it only supports sending after the node `Fulfilled`.

### Version(s)

main

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
templates:
- name: gen-random-int
retryStrategy:
limit: 11
retryPolicy: Always
backoff:
duration: '20'
factor: 2
maxDuration: 20m
affinity:
nodeAntiAffinity: {}
metrics:
prometheus:
- name: node_timeout_metrics
help: Duration gauge by name
when: '{{duration}} > 100'
gauge:
value: '{{duration}}'
realtime: true
```

### Logs from the workflow controller

```text
null
```

### Logs from in your workflow's wait container

```text
null
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the minimal workflow with the template-level Prometheus gauge using `realtime: true`, then trace how the workflow controller evaluates metrics before and after node fulfillment. Done means the `when: '{{duration}} > 100'` metric is emitted during execution rather than only after the node is fulfilled, with a regression test covering the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, prometheus
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.