cockroachdb / cockroachdb/cockroach

kvserver/load: track per-replica load statistics at different levels of granularity

Open
#107,890 0 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

We record various rated throughput and resource consumption statistics per replica:

https://github.com/cockroachdb/cockroach/blob/f867ce5d8304b22c177901602a6bfc7707d79b54/pkg/kv/kvserver/load/replica_load.go#L47-L47

These statistics are maintained at the same granularity, regardless of whether there is little, or no load.

The primary replica load statistic user, allocation decisions, does not require a high degree of accuracy for lightly loaded replicas.

It would be nice to track per-replica load statistics in less granularity, when the replica has below a configurable amount of load.

**Describe the solution you'd like**

Switch replicas to use a lower overhead data structure once the rate is below some threshold[^1], and to use the [sliding window data structure](https://github.com/cockroachdb/cockroach/blob/1d3c11efe8a5368cff3a3eec4668bf866c10952d/pkg/kv/kvserver/replicastats/replica_stats.go#L69-L69) when above.

**Additional context**

It would be beneficial to look at quantifying the current tracking overhead per-replica [^2] before beginning on the issue.

A candidate data structure for lower overhead rate tracking is [EWMA](https://github.com/VividCortex/ewma/blob/master/ewma.go).

[^1]: The configurable threshold doesn't need to be absolute (e.g. `x > 5`), it can be relative, such as tracking the top K at higher granularity; or kth percentile.
[^2]: https://github.com/cockroachdb/cockroach/pull/93823 has context from previous stat tracking consolidation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.