cockroachdb / cockroachdb/cockroach
kv: properly handle benign errors in queues
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When range descriptor changes are made as part of queue operations such as upreplication (replicate queue), splits, merges, and so on, a [`ConditionFailedError` occurs](https://github.com/cockroachdb/cockroach/blob/c8dbdeb8013c8df821525c0bb2e76eb78bfd88ec/pkg/kv/kvserver/replica_command.go#LL3249C41-L3249C41) when the range descriptor found on disk differs from what was expected, which can occur as part of normal operation when there are concurrent updates to a range descriptor. These errors are [marked with a flag](https://github.com/cockroachdb/cockroach/blob/c8dbdeb8013c8df821525c0bb2e76eb78bfd88ec/pkg/kv/kvserver/markers.go#L60) stating that replication can be retried with an up-to-date range descriptor, and are [wrapped in an error type explicitly named `benignError`](https://github.com/cockroachdb/cockroach/blob/c8dbdeb8013c8df821525c0bb2e76eb78bfd88ec/pkg/kv/kvserver/replica_command.go#L390).
Despite this, there are places on the server side of these operations where the errors are improperly wrapped (e.g. on [merge](https://github.com/cockroachdb/cockroach/blob/c8dbdeb8013c8df821525c0bb2e76eb78bfd88ec/pkg/kv/kvserver/replica_command.go#L808)), effectively dropping those markings indicating that the client can retry, as well as places in the queues themselves (e.g. in the [replicate queue](https://github.com/cockroachdb/cockroach/blob/c8dbdeb8013c8df821525c0bb2e76eb78bfd88ec/pkg/kv/kvserver/replicate_queue.go#L952)) where these flags are not considered, potentially resulting in replicas that should be reprocessed immediately getting put into purgatory and not tried again for some time.
We have seen this come up as an issue in customer escalations, as well as in [roachtest failures](https://github.com/cockroachdb/cockroach/issues/102225). As part of this issue, we should identify and fix places where this error is improperly returned from the server side and/or improperly handled on the client side in the queues.
Jira issue: CRDB-27671
Contributor guide
Assessment
This issue has not been assessed yet.