cockroachdb / cockroachdb/cockroach

kvcoord: clarify and split up DistSender errors

Open
#143,440 1 comment 0 reactions 0 assignees View on GitHub
A-kv-client branch-master C-bug O-postmortem P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently, there is a large set of errors constructed and propagated by the DistSender. Some of them have a clear meaning (like `distsender.errors.notleaseholder`). Others, like `distsender.rpc.sent.nextreplicaerror`, are a catch-all for any reason `sendToReplicas` decided to go through one more iteration of the retry loop. Not all errors that might fall into `distsender.errors.notleaseholder` are categorized by other metrics.

During investigations, it's not clear which errors are a subset of other errors. For example, when we saw this context cancelation error in a CPU profile, we couldn't easily tell which DistSender metrics are expected to account for it.

https://github.com/cockroachdb/cockroach/blob/e048c1b1f3821a7ac57c6c8ca2077084150849fa/pkg/kv/kvserver/replica_send.go#L430-L432

To make it easier to work with these metrics, we should either add more documentation in the metrics descriptions, or separate metrics out better. For the specific context cancelation error above, we could consider adding a new metric.

Jira issue: CRDB-48836

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.