apache / apache/cloudstack

Changing CPU overprovisioning factor breaks prometheus and listHosts usage metrics

未關閉
#7,591 9 則留言 0 個 reaction 已指派 1 人 已被 @RosiKyu 認領 在 GitHub 檢視
component:management-server component:prometheus no-issue-activity Severity:Minor
主要語言
Java
星號
3.1k
分支
1.4k
平均合併
6 天 19 小時
30 天內合併 PR
32

描述

##### ISSUE TYPE

* Bug Report

##### COMPONENT NAME

~~~
Prometheus Exporter
API
~~~

##### CLOUDSTACK VERSION

~~~
4.17.0
4.18.0
~~~

##### CONFIGURATION

N/A

##### OS / ENVIRONMENT

N/A

##### SUMMARY

When CPU overcommit factor is changed, the prometheus exporter metric "cloudstack_host_cpu_usage_mhz_total" as well as API response of listHosts (cpuused field) seems to be multiplied to the new overcommit factor.

The actual "used" metrics should not be affected by overcommit factor. Overcommit factor should only virtually increase the capacity the node has, and not affecting the usage metric.

##### STEPS TO REPRODUCE

~~~
1. Empty out a hypervisors from VMs, VRs, systemvms etc. So there is no virtual machines running on it.
2. Pick a virtual machine to start on that hypervisor. Before starting, note the amount of CPU cores and CPU Mhz it has, e.g two cores 500Mhz each.
3. After you have started the test virtual machine on the test hypervisor, check the Prometheus cloudstack_host_cpu_usage_mhz_total{hostname=}. It should show the CPU Mhz used on that hypervisor: cpu_number * cpu_mhz, e.g. 1000. This is the correct value.
4. Now change the cluster setting cpu.overprovisioning.factor to a new value, e.g. 4.
5. The cloudstack_host_cpu_usage_mhz_total{hostname=} now shows different value, presumably calulated by the formula: cpu_number * cpu_mhz * (new_overprovisioning_factor - old_overprovisioning_factor)
6. If you stop and start the test VM, then the cloudstack_host_cpu_usage_mhz_total goes back to normal.

Same reproduce steps can be applied to the API response of listHosts call, field cpuused.
If you start a VM, then change overprovisioning factor, the field will contain incorrect value (especially if you put ridiculously high overprovisioning factor value, such as 1000).
~~~

##### EXPECTED RESULTS

~~~
The prometheus metric cloudstack_host_cpu_usage_mhz_total and API response of listHosts call (field cpuused) should not contain overprovisioning factor in their calculation as usage reports report on real usage.
~~~

##### ACTUAL RESULTS

~~~
The metric is reported without overprovisioning factor in its calculation when a VM starts, then gets distorted when you change overprovisioning factor.
~~~

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。