kv: sustained write load across many ranges with high RTT replication can overload (shared) raft transport

High-latency links can have unexpectedly low effective throughputs due to the dynamic presented in https://github.com/cockroachdb/cockroach/issues/111238. As a result, it is possible to overload a raft transport link between two distant nodes before hitting other hardware bottlenecks first.

The purpose of this issue is to demonstrate this effect, which motivates https://github.com/cockroachdb/cockroach/issues/111238 and https://github.com/cockroachdb/cockroach/issues/111239.

### Reproduction steps:

```bash
# create a multi-region cluster
roachprod create nathan-high-rtt --nodes=4 --gce-machine-type=n2-standard-16 --clouds=gce --geo --gce-zones=us-east1-b,us-west1-b,asia-east1-a
roachprod stage nathan-high-rtt release v23.1.10
roachprod start nathan-high-rtt:1-3
```

This creates a cluster with the following latencies between nodes:

Using the BDP math presented in https://github.com/cockroachdb/cockroach/issues/111238, we would expect that a single TCP connection between `us-east1` and `asia-east1` with a default 4MB window size could theoretically reach a throughput of `1s/187ms * 4MB = 21MB/s`. `iperf` shows 14 MB/s.

```bash
# initialize a kv table with leases in us-east
roachprod run nathan-high-rtt:1 -- './cockroach workload init kv'
roachprod sql nathan-high-rtt:1 -- -e "alter database kv configure zone using constraints = copy from parent, lease_preferences = '[[+region=us-east1]]'"

# Slowly ramp up the load using 2KB writes
roachprod run nathan-high-rtt:4 -- './cockroach workload run kv --min-block-bytes=2048 --max-block-bytes=2048 --concurrency=4 --duration=1m {pgurl:1}'
roachprod run nathan-high-rtt:4 -- './cockroach workload run kv --min-block-bytes=2048 --max-block-bytes=2048 --concurrency=16 --duration=1m {pgurl:1}'
roachprod run nathan-high-rtt:4 -- './cockroach workload run kv --min-block-bytes=2048 --max-block-bytes=2048 --concurrency=64 --duration=1m {pgurl:1}'
roachprod run nathan-high-rtt:4 -- './cockroach workload run kv --min-block-bytes=2048 --max-block-bytes=2048 --concurrency=256 --duration=1m {pgurl:1}'
roachprod run nathan-high-rtt:4 -- './cockroach workload run kv --min-block-bytes=2048 --max-block-bytes=2048 --concurrency=1024 --duration=1m {pgurl:1}'
```

This initial load test looks like the following: (click here)

Notice that we scale up to the point where n1 (`us-east1`) is sending n3 (`asia-east1`) almost exactly 14MB. Beyond that point, we start seeing congestion in the raft transport, though it never fills up to the point of dropping messages. Recall that the raft transport is shared between ranges, so this would affect other communication between n1 and n3, not just on the `kv` range. However, the congestion never gets too bad because the quota pool kicks in and starts throttling all writes to the single `kv` range. Preventing a single slow follower from falling indefinitely behind is the purpose of the quota pool, though it's arguably a surprising effect that a minority quorum can impact availability, especially so quickly.

This already looks bad, but the quota pool is providing some protection. That's because we're only writing to a single range, so per-range limits like the quota pool and flow control in `etcd/raft` prevent sustained saturation of the raft transport. What if this exact same load was spread over 1000 ranges?
```bash
roachprod run nathan-high-rtt:1 -- './cockroach workload init --splits=1000 kv'
roachprod run nathan-high-rtt:4 -- './cockroach workload run kv --min-block-bytes=2048 --max-block-bytes=2048 --concurrency=1024 --duration=1m {pgurl:1}'
```

This test looks like the following: (click here)

Notice again that the n1->n3 network link only sustained about 15MB/s while the n1->n2 link sustained 20MB/s. So n3 fell further and further behind as the load ran. However, unlike the previous test, the KV table is spread across 1000 ranges, so a single range's quota pool didn't start backpressuring the workload. Instead, the load was allowed to pass through to the shared (across ranges) raft transport from n1->n3 and saturate it for the entire duration of the test. The raft transport hit its maximum size of 10k messages and then began dropping messages.

As a result, the link became effectively unusable, even for unrelated ranges (see https://github.com/cockroachdb/cockroach/issues/111239). With a queue size of 300MB and a processing rate (the network throughput) of 15MB/s, even messages from unrelated ranges that were lucky enough to accepted by the queue and not bounce off of its size limit would take over 20s to be sent out to n3. Over a sustained period of time, this could allow unrelated ranges serving a much lower rate of writes to also eventually drain their quota pool and backpressure incoming writes.

To summarize, the key ingredients to overload the raft transport between two nodes are:
- high latency
- high write throughput
- large writes
- across many ranges

Combine all four of these and the raft transport will saturate before any other hardware bottlenecks are reached.

Jira issue: CRDB-31832