metrics: pre-define telemetry datapoints for easier consumption
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
The Nomad agent does not pre-define datapoints, so each metric only appears in the API (for example when using Prometheus) during a specific window after it has been triggered. If the datapoint is not triggered again within a certain period, the datapoint will be garbage collected and will cease to appear in the API return object. In a similar manner, if a datapoint is never triggered, it will never show up in the API return.
In clusters with high throughput that utilise all features, this isn't a huge problem. In all but the largest clusters, this becomes a problem, and when setting up dashboards on idle clusters, it makes the task quite tricky.
Example: I am settings up Grafana dashboards to monitor certain Nomad subsystem as part of a project. In particular I want to look at the scheduler performance and our documentation indicates the metric `nomad.worker.invoke_scheduler.` is important. I am not currently utilising all the scheduler types, which means I cannot create graphs for all of them unless I trigger samples jobs to trigger the datapoint.
Ideally, Nomad would pre-define its metrics, or at least some, so the problems described above can be lessened.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Nomad's agent metrics API and its Prometheus representation, using nomad.worker.invoke_scheduler. as a concrete example. The issue names no files or tests, so first locate the metrics implementation and existing coverage; done means the requested datapoints appear for idle clusters without first triggering them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grafana, prometheus
- Domain
- api, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100