cockroachdb / cockroachdb/cockroach

kvcoord: After changing from multi-store to single-store nodes, abandoned replicas can get stuck in the GC queue

Open
#107,699 2 comments 0 reactions 0 assignees View on GitHub
C-bug O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

In a customer case, after changing from a multi-store to single-store environment, the system appeared to work correctly, however a few days later it experienced a number of problems with routing in DistSender. The underlying issue was the abandoned replicas were retried forever and eventually prevented other replicas from being routed.

**To Reproduce**

1) Convert a single-store system to multi-store.
2) Let it sit for a little bit to rebalance.
3) Create and delete some tables and data while it is in this configuration.
4) Temporarily disable the replicaGC queue
5) Convert it from multi-store back to single store by removing the disks gradually one at a time and letting the system rebalance data.
6) Turn the replicaGC queue back on
7) Notice the errors in the logs on nodes - they will look like:

```
kv/kvclient/kvcoord/dist_sender.go:1696 slow range RPC: have been waiting 60.96s (65 attempts) for RPC PushTxn(...) to r22298: [(n7,s29):51, (n4,s22):35, (n1,s17):53, next=54, gen=2595, sticky=1688401984.449023734,0]; resp: failed to send RPC: sending to all replicas failed; last error: store 22 was not found
```

**Expected behavior**
Dist sender should not get stuck on a replica that returns a `StoreNotFoundError`. Instead it should fail the request quickly if all stores return that error.

Jira issue: CRDB-30166

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.