cockroachdb / cockroachdb/cockroach

kvcoord: DistSender should export latency histogram

Open
#102,003 0 comments 0 reactions 0 assignees View on GitHub
C-enhancement O-testcluster T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

Inspired by 23.1 test cluster investigations.

```
W230421 13:11:40.669329 409838220 kv/kvclient/kvcoord/dist_sender.go:1678 ⋮ [T1,n8,job=‹AUTO CREATE STATS
id=858561249542242312›,create-stats-distsql] 267430 slow range RPC: have been waiting 197.97s (1 attempts
) for RPC Scan [/Table/125/1,/Table/125/1/‹6388773218069834480›), [txn: 6c8fafbb] to r1261:‹/Table/125{-/1
/6388773218069834480}› [(n8,s8):6, (n3,s3):12, (n6,s6):11, next=13, gen=123]; resp: ‹(err: ), *kvpb.S
canResponse›
```

We shouldn't see stuff like this _only_ in the logs (we have the "slow RPCs" metric but the more general point is, we should see the distribution of all latencies on top of this logging so that we know _when_ to look in the logs, too).

We have zero visibility into anything below the duration threshold for logging.

**Describe the solution you'd like**

A tasteful histogram or two :-) Noting that requests that take a long time may frequently end in errors, so we should consider a histogram for error responses too.

**Describe alternatives you've considered**

**Additional context**

Jira issue: CRDB-27211

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.