cockroachdb / cockroachdb/cockroach
kvserver: consider removing samples tracked by the splitFinder in some cases
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
The `splitFinder` uses reservoir sampling to track a list of sample keys. These samples aren't reset all that often -- they're [only reset](https://github.com/cockroachdb/cockroach/blob/9fec634ed37cc7fdd3916b8e2386bb2e63d01561/pkg/kv/kvserver/split/decider.go#L391) when:
1. When there is a load based split.
2. The rebalancing objective for the range changes.
3. There is a leaseholder change.
Notably, because samples aren't reset when a range's bounds change (i.e. it splits because of a reason other than load based splitting), it's quite possible that the best split key being tracked by the `splitFinder` isn't actually in the range bounds. See internal [escalation for an example.](https://github.com/cockroachlabs/support/issues/3227#issue-2953581615)
Additionally, any samples that have been recorded aren't decayed over time (unlike the `split.Decider.maxStatTracker`, which weights more recent QPS/CPU usage higher).
Both these issues together mean that if we're stuck with a bad sample of keys we'll end up nerfing load based splitting entirely.
----
Some ways we could improve the situation:
1. Never track sample keys that are outside a range's bounds. We could either reset the `split.Decider` on any split, not just a load based split. Alternatively, we could cull samples that no longer fall into the post split range boundary.
2. Never return a load based split key that's not within a range's boundary in `split.Decider.MaybeSplitKey`.
3. Have some notion of decaying in the `splitFinder`, such that samples from the past are eventually cleaned out. The ask here is a bit handwavy.
4. Reset the `split.Decider` if we attempted a load based split on a key outside the range's boundary.
cc @kvoli for when you're back. Also, s/o to @pav-kv for debugging a bunch of this.
Jira issue: CRDB-49369
Contributor guide
Assessment
This issue has not been assessed yet.