cockroachdb / cockroachdb/cockroach
metrics: make certain `sys` network/disk stats counters instead of gauges.
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
In the [RuntimeStatsSampler](https://github.com/cockroachdb/cockroach/blob/8f9923f549ca94232549ab31a40ef2f089af9771/pkg/server/status/runtime.go#L250), we gather various system metrics from the host on things like memory, disk, network, cpu, etc.
A few of these metrics, such as `sys_host_disk_[read|write]_bytes` and `sys_host_net_[send|recv]_bytes`, are cumulative monotonically increasing values. However, these metrics are exported as `GAUGE` metrics. This makes it difficult to calculate common functions like rate-of-change against the metrics, as reported by one customer struggling to do so in NewRelic.
**Describe the solution you'd like**
Identify gauges in the RuntimeStatusSampler that would be better represented as a counter, and migrate the metrics over.
Note that our `metric.Counter` API only allows to `.Inc(int64)`/`Count()`, while the raw measurements are cumulative. Therefore, we will need to compute the delta since the last measurement when updating the metric values. This is [already done in some areas](https://github.com/cockroachdb/cockroach/blob/8f9923f549ca94232549ab31a40ef2f089af9771/pkg/server/status/runtime.go#L480-L481) for other purposes, so we can likely piggy-back off of that work.
**Additional context**
Be sure to consider the impact on downstream metric systems of changing the metric types. To my knowledge, this would lead to a new & separate timeseries in systems like Prometheus, breaking metric continuity. But how will two metrics with the same name but different types fare in these systems? We should test the migration out by running a Prometheus instance scraping a CockroachDB node with the `gauge` metrics, and then restart the node with the metrics changed over to `counter`s and see the impact when viewing the metric in Grafana.
Be sure to backport this to ~~v22.1+~~[ v22.2+](https://github.com/cockroachdb/cockroach/issues/102729#issuecomment-1531940332). The specific customer experiencing the issue in NewRelic is on v22.1.18, but plans on upgrading to v22.2 soon.
Jira issue: CRDB-27619
Epic CRDB-32137
Contributor guide
Assessment
This issue has not been assessed yet.