cockroachdb / cockroachdb/cockroach
server: drain hangs when replicas unavailable on node
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
On 23.1 testcluster. We had a node that had subsumed replicas that didn't know about the merge yet. They were basically in a loss of quorum state, since their peers had all disappeared (been merged away).
We had disabled replica circuit breakers due to https://github.com/cockroachdb/cockroach/issues/101993, and attempted a node that had these abandoned replicas. The drain hung (without ever printing any progress to stdout) and hit `--drain-wait` after 10m.
Stack trace showed that we were stuck in because of https://github.com/cockroachdb/cockroach/blob/d6b664684d412da63b4a2224fc0a081527b800d2/pkg/kv/kvserver/store.go#L1626-L1628
On these defunct ranges there's always an (ill fated) acquisition in flight. Six goroutines that try to transfer leases were observed in
```
goroutine 407694556 [chan receive, 6 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SetDraining.func1.1.1({0x6fa51a0, 0xc1c439d7a0})
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:1632 +0x1cb
```
meaning they attached themselves to a lease acquisition request that didn't have a timeout. They should at least use a timeout themselves to avoid stalling the drain process' ability to report progress.
**To Reproduce**
I think this should reproduce when setting up a three node cluster, turning off replica circuit breakers, killing two nodes, and trying to drain the remaining nodes. It'd be interesting to also see how it behaves with the circuit breakers on (which is the default) - my expectation is that each drain poll will return, but that the outstanding work will never drop to zero, which is a reasonable failure mode.
**Expected behavior**
Add a fairly short timeout to https://github.com/cockroachdb/cockroach/blob/d6b664684d412da63b4a2224fc0a081527b800d2/pkg/kv/kvserver/store.go#L1626-L1628 because it has a tendency to latch on to ill-fated lease acquisitions. Or better, provide an ability to latch on only to lease *transfers*, but not *requests* (here we must have seen requests IMO).
Jira issue: CRDB-27210
Contributor guide
Assessment
This issue has not been assessed yet.