cockroachdb / cockroachdb/cockroach
kvserver: limit Raft ready memory usage across replicas
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When applying committed Raft entries, we pull 64 MB of commands into memory per Raft ready cycle.
https://github.com/cockroachdb/cockroach/blob/8136e8ec9847418384e0900f80d3f5320ca70b52/pkg/base/config.go#L261-L262
However, this is per range, so with e.g. `COCKROACH_SCHEDULER_CONCURRENCY` of 96 (>=12 CPU cores) we can pull up to 6 GB into memory at once. And since the scheduler concurrency is computed per store (#102838), a 10-store node can pull 60 GB into memory at once.
Additionally, this does not take into account per-entry overhead (e.g. `replicatedCmdBufNode` and `raftpb.Entry`), which can be significant in the case of small commands and has been [seen](https://cockroachlabs.slack.com/archives/C056EK7B7JQ/p1683250364610459?thread_ts=1683249502.736709&cid=C056EK7B7JQ) to amplify memory usage by a factor of 12.
We need a global memory budget for command application across all replicas on a node, which also takes into account struct overhead.
Jira issue: CRDB-27680
Contributor guide
Assessment
This issue has not been assessed yet.