cockroachdb / cockroachdb/cockroach

kvcoord: better error reporting/handling for `DistSender` after GC/split race fix in #162271

Open
#165,995 0 comments 0 reactions 0 assignees View on GitHub
A-kv-client A-kv-transactions C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

With the fix for the GC/split race introduced in #162085, the replica returns a `RangeKeyMismatchError` when a `ClearRange` request straddles range boundary. This error, when received by the `DistSender`, triggers the following code path in the file [`pkg/kv/kvclient/kvcoord/dist_sender.go:2365`](https://github.com/cockroachdb/cockroach/blob/2034816775da79c85b9872c56a494a94f458a8c9/pkg/kv/kvclient/kvcoord/dist_sender.go#L2365)

```
case *kvpb.RangeKeyMismatchError:
// Range descriptor might be out of date - evict it. This is likely the
// result of a range split. If we have new range descriptors, insert them
// instead.
for _, ri := range tErr.Ranges {
// Sanity check that we got the different descriptors. Getting the same
// descriptor and putting it in the cache would be bad, as we'd go through
// an infinite loops of retries.
if routingTok.Desc().RSpan().Equal(ri.Desc.RSpan()) {
return response{pErr: kvpb.NewError(errors.AssertionFailedf(
"mismatched range suggestion not different from original desc. desc: %s. suggested: %s. err: %s",
routingTok.Desc(), ri.Desc, pErr))}
}
}
```

And returns an `AssertionFailedError`. This assertion failure is orthogonal to the error. We should improve this error reporting to make it more pertinent to the underlying issue of the range key mismatch. Or perhaps percolate this error up the call stack and handle it appropriately in the GC package.

```
260316 22:23:14.046225 38411 15@kv/kvserver/gc/gc.go:979 ⋮ [T1,Vsystem,n2,s2,r84/2:/Table/100/‹"›{‹1d5a8c›…-‹5c4166›…}] 4100 failed to GC keys with clear range: mismatched range suggestion not different from original desc. desc: r86:/Table/100/‹"›{‹5c41661c8ae2e00b"›-‹6f3126e72b7a596a"›} [(n5,s5):1, (n2,s2):2, (n1,s1):3, next=4, gen=38, sticky=9223372036.854775807,2147483647]. suggested: r86:/Table/100/‹"›{‹5c41661c8ae2e00b"›-‹6f3126e72b7a596a"›} [(n5,s5):1, (n2,s2):2, (n1,s1):3, next=4, gen=38, sticky=9223372036.854775807,2147483647]. err: key range /Table/100/‹"2acd63b39b7de847"›-/Table/100/‹"6f3126e72b7a596a"› outside of bounds of range /Table/100/‹"5c41661c8ae2e00b"›-/Table/100/‹"6f3126e72b7a596a"›; suggested ranges: [‹desc: r86:/Table/100/"{5c41661c8ae2e00b"-6f3126e72b7a596a"} [(n5,s5):1, (n2,s2):2, (n1,s1):3, next=4, gen=38, sticky=9223372036.854775807,2147483647], lease: repl=(n2,s2):2 seq=8 start=1773699791.003977053,0 type=LeaseEpoch epo=1 min-exp=0,0 pro=1773699791.013170490,1 acq=Request, closed_timestamp_policy: LAG_BY_CLUSTER_SETTING› ‹desc: r84:/Table/100/"{1d5a8c79153a643b"-5c41661c8ae2e00b"} [(n5,s5):1, (n2,s2):2, (n1,s1):3, next=5, gen=38, sticky=9223372036.854775807,2147483647], lease: repl=(n2,s2):2 seq=8 start=1773699791.003977053,0 type=LeaseEpoch epo=1 min-exp=0,0 pro=1773699791.013170490,1 acq=Request, closed_timestamp_policy: LAG_BY_CLUSTER_SETTING›]
```

Jira issue: CRDB-61707

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.