cockroachdb / cockroachdb/cockroach

kvserver: raft OOM when catching up node with many ranges and large rows

Open
#105,338 8 comments 0 reactions 1 assignee Claimed by @pav-kv View on GitHub
C-performance O-23.2-scale-testing O-testcluster T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

While working on #103288, I spun up a 3-node `n2-highcpu-8` cluster (8 vCPUS, 8 GB memory) with a bank workload writing 10 KB rows across 35k ranges. After some time, I took down one of the nodes for about 30 minutes. When I reintroduced it to the cluster, it continually OOMed on startup, with heap profiles showing all memory usage came from Raft request decoding (likely MsgApps kept around in the unstable log). Increasing memory from 8 GB to 32 GB was not sufficient to resolve the OOMs.

Rough repro:

```
roachprod create -n 4 --gce-machine-type n2-highcpu-8 --local-ssd=false --gce-pd-volume-size 2000 grinaker-lease

roachprod start grinaker-lease:1-3

SET CLUSTER SETTING kv.bulk_io_write.concurrent_addsstable_requests = 4;

./cockroach workload init bank --rows 100000000 --ranges 35000 --payload-size 10000 --data-loader IMPORT $PGURLS
./cockroach workload run bank --rows 100000000 --batch-size 100 --payload-size 10000 --concurrency 64 $PGURLS

SET CLUSTER SETTING kv.expiration_leases_only.enabled = true;
```

Let the workload run for 20 minutes. Stop one of the nodes, keep it down for 20 minutes, restart it.

35k ranges is probably excessive here, try e.g. 20k ranges. kv0 with large rows/batches probably does the trick too. The initial import here will take 5 hours, a smaller initial dataset probably works too.

Jira issue: CRDB-28990

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.