cockroachdb / cockroachdb/cockroach

kvserver: limit Raft ready memory usage across replicas

Open
#102,840 2 comments 0 reactions 0 assignees View on GitHub
C-enhancement O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When applying committed Raft entries, we pull 64 MB of commands into memory per Raft ready cycle.

https://github.com/cockroachdb/cockroach/blob/8136e8ec9847418384e0900f80d3f5320ca70b52/pkg/base/config.go#L261-L262

However, this is per range, so with e.g. `COCKROACH_SCHEDULER_CONCURRENCY` of 96 (>=12 CPU cores) we can pull up to 6 GB into memory at once. And since the scheduler concurrency is computed per store (#102838), a 10-store node can pull 60 GB into memory at once.

Additionally, this does not take into account per-entry overhead (e.g. `replicatedCmdBufNode` and `raftpb.Entry`), which can be significant in the case of small commands and has been [seen](https://cockroachlabs.slack.com/archives/C056EK7B7JQ/p1683250364610459?thread_ts=1683249502.736709&cid=C056EK7B7JQ) to amplify memory usage by a factor of 12.

We need a global memory budget for command application across all replicas on a node, which also takes into account struct overhead.

Jira issue: CRDB-27680

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.