cockroachdb / cockroachdb/cockroach

kvserver: add roachtests for raft memory pressure

Open
#111,259 1 comment 0 reactions 1 assignee Claimed by @pav-kv View on GitHub
A-kv-replication C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

A CRDB node can host (tens of) thousands of Raft instances, which operate semi-independently. Each instance has a limited footprint, e.g. it can pull up to a certain amount of data in memory, and have a certain amount of in-flight messages. There is no node-wide resource limit for Raft, and, as a result, under certain circumstances (#73376, #102840, #105338) a node gets overloaded and OOMs. There is a work track to add memory limits and prevent these overflows.

In the meantime, there are a number of tests (such as #110764) susceptible to this issue, and failing occasionally. We should have a roachtest that reproduces such high-memory-usage scenarios more reliably, and use it to measure and guide improvements.

Jira issue: CRDB-31840

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.