Nomad Telemetry does not adhere to Prometheus "Best Practices".

Nomad version

1.7.7

Issue

Nomad Telemetry does not adhere to Prometheus "Best Practices".

"Time series that are not present until something happens are difficult to deal with, as the usual simple operations are no longer sufficient to correctly handle them. To avoid this, export a default value such as 0 for any time series you know may exist in advance."
Source: Prometheus avoid missing metrics.

Metrics should always have values. Not just when something happens. I have run into metrics that are exported from Nomad that do not continuously publish metric values.

One example is the nomad_client_allocs_oom_killed metric. But there is a different conversation around that.

Today I was trying to make a chart that displayed how many allocations were running in a given environment. Thankfully there is a metric for that: nomad_client_allocs_running. Unfortunately, values are only published when something happens.

Side note, I am not sure what the difference is between the following two metrics. However, I do know that nomad.client.allocations.running publishes values continuously as it should.

Host Metrics
- nomad.client.allocs.running         Number of running allocations   Integer   Counter
Allocation Metrics
- nomad.client.allocations.running    Number of allocations running   Integer   Gauge

Expected Result

I would expect to see continuously published values.

Actual Result

I only see values when some event happens.

Start by tracing the telemetry instrumentation that exports nomad_client_allocs_running and compare it with nomad.client.allocations.running, using the metric descriptions in the issue as the starting point. Check the Prometheus guidance on avoiding missing metrics. Done means the relevant Nomad metric continuously exposes a value, including when no event has occurred, with its existing semantics clarified or preserved.