cockroachdb / cockroachdb/cockroach

kvserver: a slow follower should not stall the quota pool

Open
#113,054 6 comments 0 reactions 0 assignees View on GitHub
A-kv-replication C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The quota pool will halt proposals when a certain amount of proposals are in flight and not yet replicated and applied on all replicas. This means that a single slow follower can stall the entire range. There are exceptions here, i.e. we ignore followers we haven't communicated with in a while, and also pause followers that are IO overloaded:

https://github.com/cockroachdb/cockroach/blob/d80a9f515fcba6ff874fde88f7cd1c42f583a59c/pkg/kv/kvserver/replica_proposal_quota.go#L172-L174

https://github.com/cockroachdb/cockroach/blob/d80a9f515fcba6ff874fde88f7cd1c42f583a59c/pkg/kv/kvserver/replica_proposal_quota.go#L217-L224

However, if the follower is live but simply slow, then it will drag the whole range down with it. We saw this happen e.g. in #113053, where the follower was stuck in an append loop, and kept rejecting MsgApps for hours, which stalled the range.

We shouldn't let a minority of followers stall the range. Instead, we should give up on them and allow progress with the remaining quorum. However, this requires the allocator to upreplicate elsewhere, since the range will otherwise stall if we lose one of the healthy replicas.

This is related to several previous discussions, and also replication admission control which we hope will replace the quota pool in the longer term. For example:

* https://github.com/cockroachdb/cockroach/issues/79215
* https://github.com/cockroachdb/cockroach/issues/75066
* https://github.com/cockroachdb/cockroach/issues/55806
* https://github.com/cockroachdb/cockroach/issues/106063

Jira issue: CRDB-32733

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.