cockroachdb / cockroachdb/cockroach

kv: properly handle benign errors in queues

Open
#102,826 0 comments 0 reactions 0 assignees View on GitHub
C-bug E-starter T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When range descriptor changes are made as part of queue operations such as upreplication (replicate queue), splits, merges, and so on, a [`ConditionFailedError` occurs](https://github.com/cockroachdb/cockroach/blob/c8dbdeb8013c8df821525c0bb2e76eb78bfd88ec/pkg/kv/kvserver/replica_command.go#LL3249C41-L3249C41) when the range descriptor found on disk differs from what was expected, which can occur as part of normal operation when there are concurrent updates to a range descriptor. These errors are [marked with a flag](https://github.com/cockroachdb/cockroach/blob/c8dbdeb8013c8df821525c0bb2e76eb78bfd88ec/pkg/kv/kvserver/markers.go#L60) stating that replication can be retried with an up-to-date range descriptor, and are [wrapped in an error type explicitly named `benignError`](https://github.com/cockroachdb/cockroach/blob/c8dbdeb8013c8df821525c0bb2e76eb78bfd88ec/pkg/kv/kvserver/replica_command.go#L390).

Despite this, there are places on the server side of these operations where the errors are improperly wrapped (e.g. on [merge](https://github.com/cockroachdb/cockroach/blob/c8dbdeb8013c8df821525c0bb2e76eb78bfd88ec/pkg/kv/kvserver/replica_command.go#L808)), effectively dropping those markings indicating that the client can retry, as well as places in the queues themselves (e.g. in the [replicate queue](https://github.com/cockroachdb/cockroach/blob/c8dbdeb8013c8df821525c0bb2e76eb78bfd88ec/pkg/kv/kvserver/replicate_queue.go#L952)) where these flags are not considered, potentially resulting in replicas that should be reprocessed immediately getting put into purgatory and not tried again for some time.

We have seen this come up as an issue in customer escalations, as well as in [roachtest failures](https://github.com/cockroachdb/cockroach/issues/102225). As part of this issue, we should identify and fix places where this error is improperly returned from the server side and/or improperly handled on the client side in the queues.

Jira issue: CRDB-27671

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.