cockroachdb / cockroachdb/cockroach
kvcoord: DistSender should export latency histogram
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
Inspired by 23.1 test cluster investigations.
```
W230421 13:11:40.669329 409838220 kv/kvclient/kvcoord/dist_sender.go:1678 ⋮ [T1,n8,job=‹AUTO CREATE STATS
id=858561249542242312›,create-stats-distsql] 267430 slow range RPC: have been waiting 197.97s (1 attempts
) for RPC Scan [/Table/125/1,/Table/125/1/‹6388773218069834480›), [txn: 6c8fafbb] to r1261:‹/Table/125{-/1
/6388773218069834480}› [(n8,s8):6, (n3,s3):12, (n6,s6):11, next=13, gen=123]; resp: ‹(err: ), *kvpb.S
canResponse›
```
We shouldn't see stuff like this _only_ in the logs (we have the "slow RPCs" metric but the more general point is, we should see the distribution of all latencies on top of this logging so that we know _when_ to look in the logs, too).
We have zero visibility into anything below the duration threshold for logging.
**Describe the solution you'd like**
A tasteful histogram or two :-) Noting that requests that take a long time may frequently end in errors, so we should consider a histogram for error responses too.
**Describe alternatives you've considered**
**Additional context**
Jira issue: CRDB-27211
Contributor guide
Assessment
This issue has not been assessed yet.