cockroachdb / cockroachdb/cockroach

kvserver,admission: sep-raft-log: revisit store admission control

Open
#156,778 1 comment 0 reactions 1 assignee Claimed by @sumeerbhola View on GitHub
A-admission-control A-kv-replication C-enhancement
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Store admission control (AC) shapes incoming writes/ingests based on various potential bottlenecks (a) flush throughput, (b) L0=>Lbase compaction bandwidth, (c) disk bandwidth (for elastic work). With physical raft log separation https://github.com/cockroachdb/cockroach/issues/16624, the Raft and State machine engines will share the same disk, but be separate wrt flushes and compactions, so the current AC behavior will need to be adjusted.

Since AC modeling and shaping operates at 15s intervals, it currently assumes that the lag from raft log write to state machine application is minimal (compared to the 15s interval). Since that lag is unaffected by raft and state machine separation, we can continue with that assumption. Similarly, we can stay with the current model where the actual shaping is applied when work is being presented for raft log writing. The changes will need to localized to how we do the modeling and how we compute the byte tokens.

Observations:
- Disk bandwidth: AC currently models the write amp for the engine, which will need to be modeled separately for the two engines. Most bandwidth consumption will be from the state machine engine: compactions, flushes. Raft engine will almost never compact (very low write amp) and most flushes will end up dropping most data (due to raft log truncation).
- L0 sub-levels: high sub-level count in the raft log engine is not detrimental to the system, so we could choose to ignore it.
- Flush throughput: Flush throughput will typically be more of a bottleneck for the state machine, since we will not be flushing much from the raft log engine. But it is simple enough to model both engine's peak flush throughput and use the minimum.

Jira issue: CRDB-56191

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.