cockroachdb / cockroachdb/cockroach

kvserver,admission: per tenant metrics in the shared kv/storage server

Open
#136,030 3 comments 0 reactions 0 assignees View on GitHub
A-admission-control A-kv C-enhancement O-postmortem O-support P-3 T-supportability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We seem to lack per-tenant metrics in the shared server. This hinders troubleshooting.
Examples of useful metrics:
- Admission WorkQueue latency histograms per tenant
- Intent resolution rate per tenant
- TODO: add more here

Per-tenant metrics need to account for the fact that over the lifetime of a shared server process it may see 1000s of active tenants. Based on past experience with multi-tenant systems, the set of active tenants is often in the 10s (on a server). The approach that works is to have in-memory metric storage only for the active ones, and hence only those get exported as timeseries. This results in 1000s of timeseries over a long interval (say a 30d interval) but for the interval of interest (say an hour) the number of timeseries is small. Some approaches I have seen in the past:
- Gauge metrics: In-memory state for points with a 0 value is GC'd.
- Cumulative metrics: One of two approaches:
- GC cumulative points that are not changing.
- Export as deltas: Deltas that are 0 are GC'd and not exported.

cc: @dhartunian

Jira issue: CRDB-44829

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.