kv: prolonged outage in globally overloaded cluster

Following a production incident, we have reproduced a prolonged outage that appears to be the result of the build-up of large send queues on every node being force-flushed and overwhelming the cluster.

As an example, in a 3 node cluster running at (over) maximum capacity, we can see that a short (<1 minute) node restart event causes a prolonged outage:

The most proximate cause appears to be that large send queues are able to build up on every node. The node restart results in force flushes. These force flushes overwhelm the disk and starve foreground traffic until the backlog of work is completed.

The queue in this case had grown to many GiB on a couple of the nodes:

The force-flushes resulting from the restart caused high IO Overload

The disk overload was extreme enough to cause write stalls:

**Additional Questions**

From the following graph it _appears_ as if the work from the force-flush was admitted at normal priority rather than as elastic work, contrary to our expectation:

**Next Steps**

- [ ] This reproduction results in an outage that also takes out the tsdb. To fully analyze it, we need to monitor the cluster via external monitoring, but we haven't added all the relevant metrics to our opentelemetry configuration for datadog. We should fix that and run this reproduction again.

**Related Issues**

- [ ] https://github.com/cockroachdb/cockroach/issues/155183
- [ ] https://github.com/cockroachdb/cockroach/pull/154960
- [ ] Actual production incident: https://github.com/cockroachlabs/support/issues/3450
- [ ] Issue discussing actual production outage: https://github.com/cockroachdb/cockroach/issues/154864

Jira issue: CRDB-55307
Epic: CRDB-58986

Start with the three-node reproduction described in the issue and review the force-flush, send-queue, and IO-overload evidence. Check related issue #155183, PR #154960, and issue #154864, then add the relevant metrics to the OpenTelemetry configuration for Datadog and rerun the reproduction; done means the cluster can be monitored externally for analysis.

Written by the indexing model from the issue text.