hashicorp / hashicorp/nomad

[Question] Telemetry documentation inaccuracies

Open
#7,773 8 comments 0 reactions 0 assignees View on GitHub
theme/docs type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

I'm running Nomad (0.10.2) with Docker task driver (19.03.8) prometheus metrics enabled.

In the telemetry docs it says:
* `nomad.client.allocs.....cpu.total_percent`: Total CPU resources consumed by the task across all cores

First of all, it seems like the docs are a bit out of date in regards to the labels & metric names, as its now changed to nomad.client.allocs.cpu.system/user/total_percent and labels, similar to how Host Metrics are documented (the post 0.7 change).

The description of the metric is confusing as well.
The `total_percent` metric can easily spike above 100%, which means it's not "across all cores" but actually a summary of. Maybe it's a language barrier on my side. I was expecting a "normalized" value here. I.e in a 4 core machine, having 1 core at 100% use leads to 25% in this metric. But in here we get 100% instead (so the max is 400%).

Because Nomad doesn't expose any "Number of cores" metric, this makes it hard to estimate how high the utilization really is. Additionally, there are no metrics that can tell how far of are we from the CPU resource limit in a similar fashion to what is being graphed on the allocation page in the nomad UI.

The memory related metrics for the allocation doesn't have similar issues as we can compute utilization by doing `nomad_client_allocs_memory_usage` / `nomad_client_allocs_memory_allocated`.
(btw `nomad_client_allocs_memory_usage` is not documented either or maybe it was renamed from `nomad_client_allocs_memory_used`).

Questions are:
a) Is this a documentation bug? Or am I just reading this incorrectly?
b) Is there any way we can graph the utilization of an allocation against its CPU resource limit?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the telemetry documentation with the current Prometheus allocation metric names and the behavior described for CPU and memory metrics. Verify whether the labels, metric names, and CPU percentage definition are accurate, and determine whether allocation utilization against its CPU limit is documented or available. Done means the telemetry documentation answers these questions accurately or clearly records the limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, prometheus
Domain
documentation, observability-sre
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.