cockroachdb / cockroachdb/cockroach

server: drain hangs when replicas unavailable on node

Open
#102,001 0 comments 0 reactions 0 assignees View on GitHub
branch-master branch-release-22.2 branch-release-23.1 C-bug O-testcluster P-2 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

On 23.1 testcluster. We had a node that had subsumed replicas that didn't know about the merge yet. They were basically in a loss of quorum state, since their peers had all disappeared (been merged away).

We had disabled replica circuit breakers due to https://github.com/cockroachdb/cockroach/issues/101993, and attempted a node that had these abandoned replicas. The drain hung (without ever printing any progress to stdout) and hit `--drain-wait` after 10m.

Stack trace showed that we were stuck in because of https://github.com/cockroachdb/cockroach/blob/d6b664684d412da63b4a2224fc0a081527b800d2/pkg/kv/kvserver/store.go#L1626-L1628

On these defunct ranges there's always an (ill fated) acquisition in flight. Six goroutines that try to transfer leases were observed in

```
goroutine 407694556 [chan receive, 6 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SetDraining.func1.1.1({0x6fa51a0, 0xc1c439d7a0})
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:1632 +0x1cb
```

meaning they attached themselves to a lease acquisition request that didn't have a timeout. They should at least use a timeout themselves to avoid stalling the drain process' ability to report progress.

**To Reproduce**

I think this should reproduce when setting up a three node cluster, turning off replica circuit breakers, killing two nodes, and trying to drain the remaining nodes. It'd be interesting to also see how it behaves with the circuit breakers on (which is the default) - my expectation is that each drain poll will return, but that the outstanding work will never drop to zero, which is a reasonable failure mode.

**Expected behavior**

Add a fairly short timeout to https://github.com/cockroachdb/cockroach/blob/d6b664684d412da63b4a2224fc0a081527b800d2/pkg/kv/kvserver/store.go#L1626-L1628 because it has a tendency to latch on to ill-fated lease acquisitions. Or better, provide an ability to latch on only to lease *transfers*, but not *requests* (here we must have seen requests IMO).

Jira issue: CRDB-27210

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.