cockroachdb / cockroachdb/cockroach
kvserver/load: track per-replica load statistics at different levels of granularity
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
We record various rated throughput and resource consumption statistics per replica:
https://github.com/cockroachdb/cockroach/blob/f867ce5d8304b22c177901602a6bfc7707d79b54/pkg/kv/kvserver/load/replica_load.go#L47-L47
These statistics are maintained at the same granularity, regardless of whether there is little, or no load.
The primary replica load statistic user, allocation decisions, does not require a high degree of accuracy for lightly loaded replicas.
It would be nice to track per-replica load statistics in less granularity, when the replica has below a configurable amount of load.
**Describe the solution you'd like**
Switch replicas to use a lower overhead data structure once the rate is below some threshold[^1], and to use the [sliding window data structure](https://github.com/cockroachdb/cockroach/blob/1d3c11efe8a5368cff3a3eec4668bf866c10952d/pkg/kv/kvserver/replicastats/replica_stats.go#L69-L69) when above.
**Additional context**
It would be beneficial to look at quantifying the current tracking overhead per-replica [^2] before beginning on the issue.
A candidate data structure for lower overhead rate tracking is [EWMA](https://github.com/VividCortex/ewma/blob/master/ewma.go).
[^1]: The configurable threshold doesn't need to be absolute (e.g. `x > 5`), it can be relative, such as tracking the top K at higher granularity; or kth percentile.
[^2]: https://github.com/cockroachdb/cockroach/pull/93823 has context from previous stat tracking consolidation.
Contributor guide
Assessment
This issue has not been assessed yet.