cockroachdb / cockroachdb/cockroach

kvserver: prevent ranges from getting too large

Open
#128,082 1 comment 0 reactions 0 assignees View on GitHub
branch-master C-bug O-postmortem O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Typically, we start back pressuring incoming writes when a range gets 2x the maximum range size, as indicated on its SpanConfig. In a recent customer escalation, we found a range had circumvented this back pressure and ballooned up to > 100GB. We speculated that this was because of the backpressure byte tolerance:

https://github.com/cockroachdb/cockroach/blob/7e4233d09edef5420b8f12d917112ce762ac1855/pkg/kv/kvserver/replica_backpressure.go#L136-L142

The intention of `kv.range.backpressure_byte_tolerance` is to ensure reducing a range's size doesn't cause unavailability. However, this approach is risky, as this comment calls out:

https://github.com/cockroachdb/cockroach/blob/7e4233d09edef5420b8f12d917112ce762ac1855/pkg/kv/kvserver/replica_backpressure.go#L44-L46

In our case, we didn't have a single command that was over this byte threshold. Instead, the theory is that because the `shouldBackpressureWrites` check is performed above latching, if enough writes slip through while the range size is less than what's permitted, and the writes would cumulatively exceed 32MB, we'd end up in the same disabled back pressure hazard the comment warns against. We should improve the status quo in a few ways:

1. Perform this check both above and below latching. We should also consider increasing the default value of `kv.range.backpressure_byte_tolerance` to be >= the max raft command size.
2. Place a hard cap on how large we'll let a range grow. This can be a configurable cluster setting that's a multiple of `MaxRangeBytes`.

cc @nvanbenschoten @sumeerbhola

Jira issue: CRDB-40790

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.