BaryoDev / BaryoDev/barakoCMS

The workflow runner emits no metrics, so "are workflows running" is only answerable by SQL

Open
#696 0 comments 0 reactions 0 assignees View on GitHub
core enhancement
Dominant language
C#
Stars
6
Forks
7
Avg merge
4h 21m
Merged PRs (30d)
307

Description

prometheus-net is in the stack and `Features/Workflows/` emits nothing to it. `grep -rn "Metrics\|Counter\|Histogram" barakoCMS/Features/Workflows/ barakoCMS/Infrastructure/Jobs/` returns no hits.

The consequence is that the two failures this subsystem is built to survive are both invisible. If the runner stops, or a provider goes down and every attempt backs off, nothing changes on any dashboard and nobody finds out until someone reports that the emails stopped. `WorkflowRunner.ExecuteAsync` is careful never to throw out of its loop, precisely so the process does not die quietly, but there is no signal that says it is alive and making progress either.

### What to emit

Counters:

- runs queued, by trigger event
- action attempts completed, by action type and outcome (`Succeeded`, `Failed`, `Unknown`, `Skipped`)

Histogram:

- attempt duration, by action type. `WorkflowActionAttempt.DurationMs` is already recorded per attempt and thrown away in aggregate.

Gauges, and these are the ones that carry the alert:

- queue depth, runs in `Pending` or `Running`
- **age of the oldest pending run**

The second gauge is the whole point. One threshold on it catches a dead runner, a provider outage, and the starvation in the sibling issue, without knowing in advance which one happened.

### Notes

- Label by tenant only if cardinality is acceptable on a large deployment. Trigger event and action type are bounded; tenant is not.
- The job queue (`Infrastructure/Jobs/`) has the same gap and the same shape of answer. Worth doing together, since `JobRecord` already carries `AttemptCount`, `NextAttemptAt` and a `DeadLettered` state and none of it is exported.
- Follow-up for barakoBrew, not this repo: the `workflow-runs` screen should sort and filter on pending age, since that is the screen an operator opens when something has stopped.

### Where I checked

Grepped `barakoCMS/Features/Workflows/` and `barakoCMS/Infrastructure/Jobs/` for metric types and found none. #691 asks for OpenTelemetry tracing, which is a different signal and does not answer "is the queue draining". Nothing open covers workflow or job metrics.

Contributor guide

Open the contributing guide

Research direction

Start with WorkflowRunner.ExecuteAsync in Features/Workflows/ and inspect WorkflowActionAttempt.DurationMs, then review the related records in Infrastructure/Jobs/. Use the listed counters, histogram, and queue gauges as the completion criteria, covering workflow and job activity without unbounded tenant labels. Confirm the metrics expose runner health and queue progress for dashboards and alerts.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, prometheus
Domain
backend, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.