cockroachdb / cockroachdb/cockroach
stepping down replication factor (RF=5 -> RF=3) caused OOMs and a storm of under-replicated ranges
Open
C-enhancement
O-25.4-scale-testing
P-3
T-kv
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The following cluster change was made at ~16:26 UTC,
```
ALTER RANGE timeseries CONFIGURE ZONE USING num_replicas=3,num_voters=3;
ALTER RANGE default CONFIGURE ZONE USING num_replicas=3,num_voters=3;
```
Shortly after several nodes OOMed and the number of under-replicated ranges jumped from ~440k to 1.5m. The chart below is in EST; the spike in under-replicated ranges can be seen ~11:26,
See the internal (slack) thread [1] for more context.
[1] https://cockroachlabs.slack.com/archives/C0A0KECF60Y/p1765820161806959
Jira issue: CRDB-58263
Contributor guide
Assessment
This issue has not been assessed yet.