hashicorp / hashicorp/nomad

metrics: pre-define telemetry datapoints for easier consumption

Open
#19,728 0 comments 0 reactions 0 assignees View on GitHub
stage/accepted theme/metrics type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

The Nomad agent does not pre-define datapoints, so each metric only appears in the API (for example when using Prometheus) during a specific window after it has been triggered. If the datapoint is not triggered again within a certain period, the datapoint will be garbage collected and will cease to appear in the API return object. In a similar manner, if a datapoint is never triggered, it will never show up in the API return.

In clusters with high throughput that utilise all features, this isn't a huge problem. In all but the largest clusters, this becomes a problem, and when setting up dashboards on idle clusters, it makes the task quite tricky.

Example: I am settings up Grafana dashboards to monitor certain Nomad subsystem as part of a project. In particular I want to look at the scheduler performance and our documentation indicates the metric `nomad.worker.invoke_scheduler.` is important. I am not currently utilising all the scheduler types, which means I cannot create graphs for all of them unless I trigger samples jobs to trigger the datapoint.

Ideally, Nomad would pre-define its metrics, or at least some, so the problems described above can be lessened.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Nomad's agent metrics API and its Prometheus representation, using nomad.worker.invoke_scheduler. as a concrete example. The issue names no files or tests, so first locate the metrics implementation and existing coverage; done means the requested datapoints appear for idle clusters without first triggering them.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grafana, prometheus
Domain
api, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.