cockroachdb / cockroachdb/cockroach
kvserver: add roachtests for raft memory pressure
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
A CRDB node can host (tens of) thousands of Raft instances, which operate semi-independently. Each instance has a limited footprint, e.g. it can pull up to a certain amount of data in memory, and have a certain amount of in-flight messages. There is no node-wide resource limit for Raft, and, as a result, under certain circumstances (#73376, #102840, #105338) a node gets overloaded and OOMs. There is a work track to add memory limits and prevent these overflows.
In the meantime, there are a number of tests (such as #110764) susceptible to this issue, and failing occasionally. We should have a roachtest that reproduces such high-memory-usage scenarios more reliably, and use it to measure and guide improvements.
Jira issue: CRDB-31840
Contributor guide
Assessment
This issue has not been assessed yet.