cockroachdb / cockroachdb/cockroach
kvcoord: better error reporting/handling for `DistSender` after GC/split race fix in #162271
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
With the fix for the GC/split race introduced in #162085, the replica returns a `RangeKeyMismatchError` when a `ClearRange` request straddles range boundary. This error, when received by the `DistSender`, triggers the following code path in the file [`pkg/kv/kvclient/kvcoord/dist_sender.go:2365`](https://github.com/cockroachdb/cockroach/blob/2034816775da79c85b9872c56a494a94f458a8c9/pkg/kv/kvclient/kvcoord/dist_sender.go#L2365)
```
case *kvpb.RangeKeyMismatchError:
// Range descriptor might be out of date - evict it. This is likely the
// result of a range split. If we have new range descriptors, insert them
// instead.
for _, ri := range tErr.Ranges {
// Sanity check that we got the different descriptors. Getting the same
// descriptor and putting it in the cache would be bad, as we'd go through
// an infinite loops of retries.
if routingTok.Desc().RSpan().Equal(ri.Desc.RSpan()) {
return response{pErr: kvpb.NewError(errors.AssertionFailedf(
"mismatched range suggestion not different from original desc. desc: %s. suggested: %s. err: %s",
routingTok.Desc(), ri.Desc, pErr))}
}
}
```
And returns an `AssertionFailedError`. This assertion failure is orthogonal to the error. We should improve this error reporting to make it more pertinent to the underlying issue of the range key mismatch. Or perhaps percolate this error up the call stack and handle it appropriately in the GC package.
```
260316 22:23:14.046225 38411 15@kv/kvserver/gc/gc.go:979 ⋮ [T1,Vsystem,n2,s2,r84/2:/Table/100/‹"›{‹1d5a8c›…-‹5c4166›…}] 4100 failed to GC keys with clear range: mismatched range suggestion not different from original desc. desc: r86:/Table/100/‹"›{‹5c41661c8ae2e00b"›-‹6f3126e72b7a596a"›} [(n5,s5):1, (n2,s2):2, (n1,s1):3, next=4, gen=38, sticky=9223372036.854775807,2147483647]. suggested: r86:/Table/100/‹"›{‹5c41661c8ae2e00b"›-‹6f3126e72b7a596a"›} [(n5,s5):1, (n2,s2):2, (n1,s1):3, next=4, gen=38, sticky=9223372036.854775807,2147483647]. err: key range /Table/100/‹"2acd63b39b7de847"›-/Table/100/‹"6f3126e72b7a596a"› outside of bounds of range /Table/100/‹"5c41661c8ae2e00b"›-/Table/100/‹"6f3126e72b7a596a"›; suggested ranges: [‹desc: r86:/Table/100/"{5c41661c8ae2e00b"-6f3126e72b7a596a"} [(n5,s5):1, (n2,s2):2, (n1,s1):3, next=4, gen=38, sticky=9223372036.854775807,2147483647], lease: repl=(n2,s2):2 seq=8 start=1773699791.003977053,0 type=LeaseEpoch epo=1 min-exp=0,0 pro=1773699791.013170490,1 acq=Request, closed_timestamp_policy: LAG_BY_CLUSTER_SETTING› ‹desc: r84:/Table/100/"{1d5a8c79153a643b"-5c41661c8ae2e00b"} [(n5,s5):1, (n2,s2):2, (n1,s1):3, next=5, gen=38, sticky=9223372036.854775807,2147483647], lease: repl=(n2,s2):2 seq=8 start=1773699791.003977053,0 type=LeaseEpoch epo=1 min-exp=0,0 pro=1773699791.013170490,1 acq=Request, closed_timestamp_policy: LAG_BY_CLUSTER_SETTING›]
```
Jira issue: CRDB-61707
Contributor guide
Assessment
This issue has not been assessed yet.