cockroachdb / cockroachdb/cockroach
kvserver,admission: per tenant metrics in the shared kv/storage server
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
We seem to lack per-tenant metrics in the shared server. This hinders troubleshooting.
Examples of useful metrics:
- Admission WorkQueue latency histograms per tenant
- Intent resolution rate per tenant
- TODO: add more here
Per-tenant metrics need to account for the fact that over the lifetime of a shared server process it may see 1000s of active tenants. Based on past experience with multi-tenant systems, the set of active tenants is often in the 10s (on a server). The approach that works is to have in-memory metric storage only for the active ones, and hence only those get exported as timeseries. This results in 1000s of timeseries over a long interval (say a 30d interval) but for the interval of interest (say an hour) the number of timeseries is small. Some approaches I have seen in the past:
- Gauge metrics: In-memory state for points with a 0 value is GC'd.
- Cumulative metrics: One of two approaches:
- GC cumulative points that are not changing.
- Export as deltas: Deltas that are 0 are GC'd and not exported.
cc: @dhartunian
Jira issue: CRDB-44829
Contributor guide
Assessment
This issue has not been assessed yet.