cockroachdb / cockroachdb/cockroach
kvcoord: clarify and split up DistSender errors
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Currently, there is a large set of errors constructed and propagated by the DistSender. Some of them have a clear meaning (like `distsender.errors.notleaseholder`). Others, like `distsender.rpc.sent.nextreplicaerror`, are a catch-all for any reason `sendToReplicas` decided to go through one more iteration of the retry loop. Not all errors that might fall into `distsender.errors.notleaseholder` are categorized by other metrics.
During investigations, it's not clear which errors are a subset of other errors. For example, when we saw this context cancelation error in a CPU profile, we couldn't easily tell which DistSender metrics are expected to account for it.
https://github.com/cockroachdb/cockroach/blob/e048c1b1f3821a7ac57c6c8ca2077084150849fa/pkg/kv/kvserver/replica_send.go#L430-L432
To make it easier to work with these metrics, we should either add more documentation in the metrics descriptions, or separate metrics out better. For the specific context cancelation error above, we could consider adding a new metric.
Jira issue: CRDB-48836
Contributor guide
Assessment
This issue has not been assessed yet.