cockroachdb / cockroachdb/cockroach
kvserver: prevent ranges from getting too large
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
Typically, we start back pressuring incoming writes when a range gets 2x the maximum range size, as indicated on its SpanConfig. In a recent customer escalation, we found a range had circumvented this back pressure and ballooned up to > 100GB. We speculated that this was because of the backpressure byte tolerance:
https://github.com/cockroachdb/cockroach/blob/7e4233d09edef5420b8f12d917112ce762ac1855/pkg/kv/kvserver/replica_backpressure.go#L136-L142
The intention of `kv.range.backpressure_byte_tolerance` is to ensure reducing a range's size doesn't cause unavailability. However, this approach is risky, as this comment calls out:
https://github.com/cockroachdb/cockroach/blob/7e4233d09edef5420b8f12d917112ce762ac1855/pkg/kv/kvserver/replica_backpressure.go#L44-L46
In our case, we didn't have a single command that was over this byte threshold. Instead, the theory is that because the `shouldBackpressureWrites` check is performed above latching, if enough writes slip through while the range size is less than what's permitted, and the writes would cumulatively exceed 32MB, we'd end up in the same disabled back pressure hazard the comment warns against. We should improve the status quo in a few ways:
1. Perform this check both above and below latching. We should also consider increasing the default value of `kv.range.backpressure_byte_tolerance` to be >= the max raft command size.
2. Place a hard cap on how large we'll let a range grow. This can be a configurable cluster setting that's a multiple of `MaxRangeBytes`.
cc @nvanbenschoten @sumeerbhola
Jira issue: CRDB-40790
Contributor guide
Assessment
This issue has not been assessed yet.