kv: clients without retries backoffs can cause metastable failure

**Describe the problem**

In situations where clients set low SQL timeouts and retry without backoff, we can enter a state of metastable failure where the only way out is to completely stop the workload and then gradually restart it.

**To Reproduce**
Use a modified version of the workload tool which will retry errors when `--tolerate-errors` is set rather than just ignoring them. Note the different binary that is put on node 13 which has this behavior.

Create a 13 node cluster (12 nodes plus workload)

```
roachprod create -n 13 --gce-machine-type n2-standard-16 $CLUSTER
roachprod stage $CLUSTER:1-12 release v23.1.17
roachprod put $CLUSTER:13 artifacts/cockroach
roachprod start $CLUSTER:1-12
roachprod ssh $CLUSTER:1 "./cockroach workload init kv $(roachprod pgurl $CLUSTER:1) --splits 1000"
```

Set up the SQL user and permissions correctly
```
USE kv;
CREATE USER testuser;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA PUBLIC TO testuser;
ALTER USER testuser SET statement_timeout='250ms';
ALTER RANGE default CONFIGURE ZONE USING gc.ttlseconds = 600;
```

2x - run this command - note that the cluster runs at ~15% CPU usage (ideally we could run this once, but sometimes it fails to start).
```
roachprod ssh $CLUSTER:13 "./cockroach-short workload run kv $(roachprod pgurl $CLUSTER:1-6 | sed 's/root/testuser/g') --concurrency=50000 --max-rate=40000 --retry-errors=0ns --ramp=10s"
```

Let it run for ~1 minutes to generate some data.
Add a write heavy workload to a different DB for a few seconds to create LSM inversion.
```
roachprod ssh $CLUSTER:13 "./cockroach workload run kv $(roachprod pgurl $CLUSTER:1-12) --tolerate-errors --concurrency=1000 --max-block-bytes=1000000 --db=kv2 --drop --init --splits=100 --max-ops=5000"
```

Notice that the system enters a failure state where the CPU is pegged and it it only processing a fraction of the number of QPS it was before.

Stop the workload jobs, wait 10 seconds and restart it. Notice that now the cluster is stable again and handling the workload without issue.

**Expected behavior**
Ideally there would be no errors that occur during this test. Given that older versions of the software hit errors due to overload during index creation, the errors are not surprising, but the non-recovery of the system is.

**Additional data / screenshots**
Timeline
* 17:29 - Cluster started with the 10K ops
* 17:30:20 - Create index - takes ~1.5s.
* 17:32 - stop workload, wait 10s and restart

**Environment:**
CRDB 23.1.17, see commands above for exact configuration.

**Additional context**
We have seen customers with similar configurations and setups that have hit this issue.

Jira issue: CRDB-38280