cockroachdb / cockroachdb/cockroach
kv: allow attaching structured stats to all batch responses
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
To better observe stats from the KV layer on a per-request basis, it would be desirable to be able to return statistics that describe the execution of individual requests (Scan, Put, etc) or at least at the level of a BatchRequest (i.e. combined stats across the request). With an easily extended mechanism, KV or even other teams (Obs) could easily add new stats of interest (such as various wait times or new counters) and build higher-level observability systems.
We have a few mechanisms that do "something along those lines":
If and only if a trace span is present (i.e. the request is being traced), contention emits a structured ContentionEvent in the span, which the higher layers can introspect:
https://github.com/cockroachdb/cockroach/blob/881d2d7ef6bbd88beeb44636e1d1812c41b803aa/pkg/kv/kvserver/spanlatch/manager.go#L630-L649
`roachpb.ScanStats` are similarly collected (for the entire BatchRequest) only when there is a trace span:
https://github.com/cockroachdb/cockroach/blob/1f98db6b78f3236cae477c66e0498321b5bc7e73/pkg/kv/kvserver/replica_evaluate.go#L280-L293
and it's currently scoped to `ScanStats`, i.e. stats related to scans (and possibly populated only in some of the request types, even though others may also perform storage-level scans).
It's worth looking into ways to lift the limitation that a trace span need to be present (this has considerable overhead) and to recast `ScanStats` as an always-on (or at least less expensive) more easily extendable primitive.
Related internal references:
- [this email thread](https://groups.google.com/a/cockroachlabs.com/g/notes-eng/c/651dF9pTJH4/m/Pba847ZqAQAJ?utm_medium=email&utm_source=footer) (though there was some off-list discussion that some of us may have in their inbox)
- https://cockroachlabs.slack.com/archives/C0959KWLJQ2/p1759844004215359 (this is a DM so only some of us can see it)
Jira issue: CRDB-55205
Contributor guide
Research direction
Start by reading pkg/kv/kvserver/spanlatch/manager.go around the ContentionEvent code and pkg/kv/kvserver/replica_evaluate.go around ScanStats. Trace how these values depend on a span and how BatchRequest responses are assembled; done means an extensible structured-stats mechanism covers batch responses without requiring tracing, with behavior validated by appropriate KV tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100