kv: split failed while applying backpressure

There is a long-standing, known issue involving `split failed while applying backpressure`.

History on this issue: #52292, #102284, #47325, #68320, #139320, #156525. We recently ran into it again in https://github.com/cockroachdb/cockroach/issues/160011.

For this issue we ran into specifically, https://github.com/cockroachdb/cockroach/issues/160011, our current hypothesis is that the reason why ranges can grow into 8GiB is caused by #128082. Once a range exceeds the split backpressure threshold, the backpressure mechanism stops applying until the hard cap (8 GiB by default) is reached. With bursty write traffic, writes can pass check at the layer above latches and then push the ranges well past the threshold (range_max_bytes * kv.range.backpressure_range_size_multiplier + kv.range.backpressure_byte_tolerance). Once ranges reach this regime, backpressure stops applying until the hard cap. This matches our observations, where the largest ranges we observed in the bad runs have 8 GiB. The relevant improvement is proposed in #129452 (not yet merged at the time of this comment), which checks whether writes should be backpressured at the layer below latching.

Next steps / follow-ups from this investigation:

- Add a roachtest repro for this issue: https://github.com/cockroachdb/cockroach/issues/161433
- Test the patch in #129452 (or set kv.range.backpressure_byte_tolerance to a very large value) and evaluate whether behavior improves.

Observed symptoms:
- Splits unable to keep up under bursty write workloads, with splits retrying several times due to timeout and slow replica grpc:
```
queue processing resulted in non-benign error: operation "‹split› queue process replica 103" timed out after 1m0.001s (given timeout 1m0s): split at key /Table/113/1/‹163›/‹68212› failed: aborted in DistSender: result is ambiguous: context deadline exceeded
slow range RPC: have been waiting 70.61s (1 attempts) for RPC ConditionalPut [/Table/113/1/‹120›/‹1›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹2›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹3›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹4›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹5›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹6›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹7›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹8›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹9›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹10›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹11›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹12›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹13›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹14›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹15›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹16›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹17›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹18›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹19›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹20›/‹0›],... 999 skipped ..., ConditionalPut [/Table/113/1/‹120›/‹1020›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹1021›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹1022›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹1023›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹1024›/‹0›], [txn: f1badeaf], [can-forward-ts] to r103:/{Table/111/3-Max} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=12, sticky=1766120740.365125063,0]; resp: (err: split failed while applying backpressure to ConditionalPut [/Table/113/1/‹120›/‹1›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹2›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹3›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹4›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹5›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹6›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹7›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹8›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹9›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹10›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹11›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹12›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹13›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹14›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹15›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹16›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹17›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹18›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹19›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹20›/‹0›],... 999 skipped ..., ConditionalPut [/Table/113/1/‹120›/‹1020›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹1021›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹1022›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹1023›/‹0›], ConditionalPut [/Table/113/1/‹120›/‹1024›/‹0›], [txn: f1badeaf], [can-forward-ts] on range r103:/{Table/111/3-Max} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=12]: operation "‹split› queue process replica 103" timed out after 1m0.001s (given timeout 1m0s): split at key /Table/113/1/‹163›/‹68212› failed: aborted in DistSender: result is ambiguous: context deadline exceeded: "sql txn" meta={id=f1badeaf
```
- Both “good” and “bad” clusters showed: high replication latency (~10s), slow latches, blocked replication streams, dropped proposals due to exceeding uncommitted entry size, failed splits due to backpressure, and ranges unable to split.
- The key difference is that the bad cluster allowed ranges to grow to 8 GiB, while the good cluster sustained the overload without hitting the hard cap.
- Both clusters had high goroutine scheduling latency (p99.9 ~30 ms), with the bad cluster slightly worse and exhibiting more slow latches.

Good:

Bad:

Jira issue: CRDB-58880