cockroachdb / cockroachdb/cockroach

kv: node crashes with OOM with elastic only AC

Open
#139,319 1 comment 0 reactions 0 assignees View on GitHub
A-kv-replication branch-release-24.3 C-bug O-perturbation T-admission-control
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

If `admission.kv.bulk_only.enabled` AC is set to true, a heavy workload which disproportionately targets one node cause that node to OOM even in medium or high memory configurations.

This can happen during the setup phase of the the index backfill perturbation test.

**To Reproduce**

This can be reproduced with the index backfill perturbation test with the following command:

```
PERTURBATION_OVERRIDE=numNodes=30,perturbationDuration=30m,acMode=elasticOnlyBoth roachtest run --cpu-quota 1024 perturbation/full/backfill
```

Alternatively this can be done manually with the following commands:
```
roachprod create --gce-machine-type n2-standard-16 --gce-local-ssd-count 2 -n 32 $CLUSTER
roachprod put $CLUSTER artifacts/cockroach
roachprod start --store-count 2 $CLUSTER:1-30
roachprod ssh $CLUSTER:1 "./cockroach workload init kv --splits 10000 {pgurl:1}"
roachprod sql $CLUSTER:1
> ALTER DATABASE kv CONFIGURE ZONE USING constraints='{"+node30":1}', lease_preferences='[[-node30]]', num_replicas=3
> SET CLUSTER SETTING kvadmission.flow_control.mode='apply_to_elastic';

# Wait until all the replicas are in place - there should be 10,000 on n30.

roachprod ssh $CLUSTER:31-32 "./cockroach workload run kv --max-block-bytes=1024 --concurrency=1000 {pgurl:1-29}
```
The reason for the 10K ranges is to prevent replica pausing or the quota pool from significantly slowing down replication to n30. The fact that this doesn't occur with lower replica counts is a good sign because it means those mechanisms are working as intended to prevent OOMs

Note that n30 will crash with an OOM after ~5-6 minutes after exhausting >55GB of RAM.

The memory is used by RaftMessageBatchRequest.Unmarshall.

**Expected behavior**
The node should not OOM.

**Additional data / screenshots**

**Environment:**
- CockroachDB version: master

**Additional context**
We have seen support cases with OOMs due to index backfills. There may be a few separate causes, but it is good to have some easy reproductions. This is expected to be less of an issue once 25.1 comes out and apply_to_all is the default AC mode. This bug is mostly useful for tracking a way to cause the memory issue to occur as it is already addressed.

Jira issue: CRDB-46598

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.