cockroachdb / cockroachdb/cockroach
kvserver: a slow follower should not stall the quota pool
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The quota pool will halt proposals when a certain amount of proposals are in flight and not yet replicated and applied on all replicas. This means that a single slow follower can stall the entire range. There are exceptions here, i.e. we ignore followers we haven't communicated with in a while, and also pause followers that are IO overloaded:
https://github.com/cockroachdb/cockroach/blob/d80a9f515fcba6ff874fde88f7cd1c42f583a59c/pkg/kv/kvserver/replica_proposal_quota.go#L172-L174
https://github.com/cockroachdb/cockroach/blob/d80a9f515fcba6ff874fde88f7cd1c42f583a59c/pkg/kv/kvserver/replica_proposal_quota.go#L217-L224
However, if the follower is live but simply slow, then it will drag the whole range down with it. We saw this happen e.g. in #113053, where the follower was stuck in an append loop, and kept rejecting MsgApps for hours, which stalled the range.
We shouldn't let a minority of followers stall the range. Instead, we should give up on them and allow progress with the remaining quorum. However, this requires the allocator to upreplicate elsewhere, since the range will otherwise stall if we lose one of the healthy replicas.
This is related to several previous discussions, and also replication admission control which we hope will replace the quota pool in the longer term. For example:
* https://github.com/cockroachdb/cockroach/issues/79215
* https://github.com/cockroachdb/cockroach/issues/75066
* https://github.com/cockroachdb/cockroach/issues/55806
* https://github.com/cockroachdb/cockroach/issues/106063
Jira issue: CRDB-32733
Contributor guide
Assessment
This issue has not been assessed yet.