cockroachdb / cockroachdb/cockroach

kvserver: very slow replicaGC for subsumed replicas

Open
#102,000 2 comments 0 reactions 0 assignees View on GitHub
C-bug O-23.2-scale-testing O-testcluster T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

From 23.1 testcluster. We got a node into a state where it had >3k replicas which were all merged away but the node hadn't realized (it was down for a bit, which is likely related).

A replica that was subsumed needs to wait for its left neighbor to either disappear or execute the merge. I think we had a long cascade of replicas all waiting for their respective left neighbors:

r1 < r2 < r3 < ... < r3000

It was difficult to trigger replicaGC for this case, because you need to do it in the right order. Additionally, can't use the SQL builtin, because these ranges by definition no longer exist in `ranges_no_leases` (the ranges were merged away).

These ranges all have their circuit breakers tripped, which means they're a confounder for using that metric usefully. Besides, they could block requests that erroneously get routed to them based on stale DistSender caches, though that should be a somewhat lesser concern since these caches would get updated quickly as the circuit breakers prevent these requests from hanging (once they engage).

**To Reproduce**

Unclear - the testcluster was running tpc-e on a 2h timer, so there was lots of split/scatter stuff going on, and the cluster was in a pretty bad state for several days.

** Desired behavior **

replicaGC is more snappy. Ideally we break the dependency on the left-neighbor, though that is likely tricky since merge correctness hinges on it. But, we could say that replicaGC for range N, if it detects a merge, queues the leftmost adjacent local replica for replicaGC; this will then knock out one replica with each replicaGC invoked, so the knot should loosen much faster.

We could also make replicaGC timings more aggressive, and check if our existing heuristics are lacking in this scenario. My expectation would be that *all* such ranges would've replicaGC'ed themselves quickly, since they would be rejected by their former peers. Perhaps subsumed ranges don't have the benefit of receiving a `ReplicaTooOldError` (since the peers don't even have any replica any more!); maybe `RangeNotFound` should also expedite replicaGC.

x-ref https://github.com/cockroachdb/cockroach/issues/101999

Jira issue: CRDB-27209

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.