[Question] Telemetry documentation inaccuracies
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
I'm running Nomad (0.10.2) with Docker task driver (19.03.8) prometheus metrics enabled.
In the telemetry docs it says:
* `nomad.client.allocs.....cpu.total_percent`: Total CPU resources consumed by the task across all cores
First of all, it seems like the docs are a bit out of date in regards to the labels & metric names, as its now changed to nomad.client.allocs.cpu.system/user/total_percent and labels, similar to how Host Metrics are documented (the post 0.7 change).
The description of the metric is confusing as well.
The `total_percent` metric can easily spike above 100%, which means it's not "across all cores" but actually a summary of. Maybe it's a language barrier on my side. I was expecting a "normalized" value here. I.e in a 4 core machine, having 1 core at 100% use leads to 25% in this metric. But in here we get 100% instead (so the max is 400%).
Because Nomad doesn't expose any "Number of cores" metric, this makes it hard to estimate how high the utilization really is. Additionally, there are no metrics that can tell how far of are we from the CPU resource limit in a similar fashion to what is being graphed on the allocation page in the nomad UI.
The memory related metrics for the allocation doesn't have similar issues as we can compute utilization by doing `nomad_client_allocs_memory_usage` / `nomad_client_allocs_memory_allocated`.
(btw `nomad_client_allocs_memory_usage` is not documented either or maybe it was renamed from `nomad_client_allocs_memory_used`).
Questions are:
a) Is this a documentation bug? Or am I just reading this incorrectly?
b) Is there any way we can graph the utilization of an allocation against its CPU resource limit?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the telemetry documentation with the current Prometheus allocation metric names and the behavior described for CPU and memory metrics. Verify whether the labels, metric names, and CPU percentage definition are accurate, and determine whether allocation utilization against its CPU limit is documented or available. Done means the telemetry documentation answers these questions accurately or clearly records the limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, prometheus
- Domain
- documentation, observability-sre
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100