cockroachdb / cockroachdb/cockroach

kvflowcontrol,admission: use flow control during raft log catchup post node-restart

Open
#98,710 3 comments 0 reactions 0 assignees View on GitHub
A-admission-control C-enhancement O-support P-3 T-admission-control
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

We've seen in write-heavy workloads that node restarts can result in LSM inversion due to a rapid onset of raft log catchup appends. This problem was touched on recently in https://github.com/cockroachdb/cockroach/issues/96521 + https://github.com/cockroachdb/cockroach/issues/95159 -- those issues amounted to 23.1 changes to avoid an immediate transfer of leases to newly-restarted nodes until their LSM is healthier, in order to stave off latency impact for leaseholder traffic. But it's possible to invert the LSM, which affects non-leaseholder traffic.

This issue proposes using the general flow control mechanism we're introducing in #95563 to pace the rate of catchup raft log appends to prevent LSM inversion entirely. With such a mechanism, we'd be able to transfer leases immediately to newly restarted nodes without lease-holder impact, and also avoid latency impact on follower traffic. https://github.com/cockroachdb/cockroach/issues/80607 is slightly related -- we could apply flow tokens to raft snapshots too to cover the general case of "catchup write traffic".

**Describe the solution you'd like**

https://github.com/cockroachdb/cockroach/blob/b84f10c0724f75eaf1ab00ebbcf60cd0d8607612/pkg/kv/kvserver/kvflowcontrol/doc.go#L310-L328

https://github.com/cockroachdb/cockroach/blob/b84f10c0724f75eaf1ab00ebbcf60cd0d8607612/pkg/kv/kvserver/kvflowcontrol/doc.go#L353-L375

This is also touched on here: https://reviewable.io/reviews/cockroachdb/cockroach/96642#-NOF0fl20XuBrK4Ywlzp.

Jira issue: CRDB-25460

Contributor guide

Open the contributing guide

Research direction

Start with pkg/kv/kvserver/kvflowcontrol/doc.go, especially lines 310-328 and 353-375, then read the flow-control work in #95563 and the linked review discussion. The goal is to define and implement pacing for catchup raft log appends so node restarts do not cause LSM inversion; no specific implementation file or test is named.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.