kvflowcontrol: pace force-flush to not cause write stalls

Quorum replication flow control (QRFC), layered over admission control (AC), couples physical work and admission close enough that raft entries are not sent to stores that do not have the capacity to admit them and perform the (physical work of) raft append and state machine application.

An exception is QRFC doing a force-flush, which ignores the store overload when doing the physical work, and only does logical queueing (as low priority) to delay the token return. Doing the physical work quickly is necessary to restore quorum quickly. Treating the send-queue work (even in the force-flush case) as low-priority is important to not cause delays to new regular work that that store is evaluating as a leaseholder. Many force-flushes are to a few ranges, so this behavior works fine.

However, in some situations where there is a massive number of send-queue bytes built up to a store, and all of these send-queues get flushed, this lack of pacing can cause massive write stalls (due to the inability of memtable flush to keep up) and read-amp in the 100s (which can take 10s of minutes to recover from). This is discussed in https://github.com/cockroachdb/cockroach/issues/154864. In general we want the operator to notice the massive send-queue backlog before the force-flush and take corrective action. Here we focus on the narrower technical problem of not overwhelming the store with these force-flushes, since overwhelming the store results in other problems e.g. store liveness failing, and lease losses. The solution outlined below will not prevent an outage due to not having quorum until the massive force-flushes complete. But it should prevent widepsread loss of leases, because store liveness support cannot write to Pebble. Which should reduce the duration of the outage. Additionally, this should provide better pacing for moderately sized force flushes and prevent them from causing write stalls and latency hiccups.

The solution space is constrained by the fact that these force flush bytes have to queue behind regular work in AC, so introducing another token pool that gets replenished on AC admission will not work -- the token pool will become empty and the force flush will stall before quorum is restored. Instead, we need (a) a shaping behavior where the receiver granting permission for more work is not tied to (the typical) AC admission, and (b) a way of signaling to the sender. We note that we already have a way of signaling shaping to the sender based on when the physical work is done, since the sender used RaftMaxInflightBytes to limit how much outstanding physical work is in progress. We could reuse that for (b).

For (a) we need a way of deciding when to do the physical work. We want to avoid capacity estimation and translation of raft entry bytes to L0 bytes (like AC does), since it adds yet another token bucket that is layered above AC queuing (and makes it harder to reason about). Instead, we adopt a simpler "wait until below some threshold" which is undifferentiated and does not need to do byte token estimation, and layer it above AC queuing:
- Wait signal: wait until memtable count drops below some fraction of the write-stall threshold (say 50%, which means 8 memtables). We may additionally condition this on some high read-amp threshold, say 50.
- The waiting will happen in the raftReceiveQueue: That is, it does not tie up a raft scheduler goroutine. Only MsgApps sent in a RaftMessageRequest with LowPriorityOverride=true will wait. The waiting happens if all RaftMessageRequests that are MsgApps and have one or more Entry have LowPriorityOverride set to true. A burst of messages can stop waiting when dropping below the wait threshold -- we will adjust the wait threshold to account for this, or have some mechanism like AC's grant chaining to apply some back pressure.

Sender behavior: The normal RaftMaxInflightBytes (default to 32MiB) is per range. With 1000s of ranges force-flushing at the same time, this can cause an OOM in the receiver. For force-flushing we will introduce a per stream "token pool" that is subtracted from when sending and added to when the bytes are no longer in-flight. This will be large, say 512MiB, since it only exists to lower the probability of an OOM on the receiver.

Jira issue: CRDB-55309
Epic: CRDB-58986

Start by tracing the raftReceiveQueue handling for MsgApps with LowPriorityOverride=true and the sender's RaftMaxInflightBytes behavior. Review how force-flush work interacts with admission control, then evaluate the proposed wait signal and per-stream token pool. Done means force-flushes are paced below the memtable threshold without tying up raft scheduler goroutines or overwhelming receivers.

Written by the indexing model from the issue text.