cockroachdb / cockroachdb/cockroach
kvserver: raft OOM when catching up node with many ranges and large rows
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
While working on #103288, I spun up a 3-node `n2-highcpu-8` cluster (8 vCPUS, 8 GB memory) with a bank workload writing 10 KB rows across 35k ranges. After some time, I took down one of the nodes for about 30 minutes. When I reintroduced it to the cluster, it continually OOMed on startup, with heap profiles showing all memory usage came from Raft request decoding (likely MsgApps kept around in the unstable log). Increasing memory from 8 GB to 32 GB was not sufficient to resolve the OOMs.
Rough repro:
```
roachprod create -n 4 --gce-machine-type n2-highcpu-8 --local-ssd=false --gce-pd-volume-size 2000 grinaker-lease
roachprod start grinaker-lease:1-3
SET CLUSTER SETTING kv.bulk_io_write.concurrent_addsstable_requests = 4;
./cockroach workload init bank --rows 100000000 --ranges 35000 --payload-size 10000 --data-loader IMPORT $PGURLS
./cockroach workload run bank --rows 100000000 --batch-size 100 --payload-size 10000 --concurrency 64 $PGURLS
SET CLUSTER SETTING kv.expiration_leases_only.enabled = true;
```
Let the workload run for 20 minutes. Stop one of the nodes, keep it down for 20 minutes, restart it.
35k ranges is probably excessive here, try e.g. 20k ranges. kv0 with large rows/batches probably does the trick too. The initial import here will take 5 hours, a smaller initial dataset probably works too.
Jira issue: CRDB-28990
Contributor guide
Assessment
This issue has not been assessed yet.