Performance degradation during TPC-C run

Hi,

I work on a performance survey. I try to execute TPC-C and it fails, when number of warehouses is above 12K (CPU and disks are not saturated though). However, I would expect CRDB to handle around 15-20K.

TPC-C run exits with either error:
```
Error: error in delivery: ERROR: inbox communication error: rpc error: code = Canceled desc = context canceled (SQLSTATE 58C01)
```
or
```
Error: error in delivery: ERROR: rpc error: code = Unavailable desc = error reading from server: read tcp [2a02:6b8:c34:14:0:1354:eb1f:2ca6]:37234->[2a02:6b8:c34:14:0:1354:eb1f:2962]:26258: use of closed network connection (SQLSTATE XXUUU)
```

Some time ago I had reported this issue in Slack and the conclusion was that CRDB was overloaded. Now, I try the newer version of CRDB.

**Setup.**

* 3 machines, each has 128 cores (2 intel xeon 6338 2.00GHz, HT enabled)
* 512 GB RAM
* 4 NVMe disks (ext4 with "rw,noatime,nodiratime,nobarrier")
* Ubuntu 20.04.3 LTS
* transparent huge pages are enabled.

On each machine I start 4 CRDB instances (v23.2.2). Each in its own NUMA-aligned taskset. Here is an example:
```bash
/opt/cockroach/cockroach start --insecure --listen-addr=:26260 --http-addr=:8083 --join=host:126257,host2:26257,host3:26257 --locality=region=us-west-1 --store /opt/cockroach/data/nvme3n1p2 --cache=37GB --max-sql-memory=37GB
```

**TPC-C run.**

TPC-C import:
```bash
cockroach workload fixtures import tpcc --warehouses=13000 'postgres://root@$COCKROACH_HOST:26257?sslmode=disable
```

TPC-C run:
```bash
cockroach workload run tpcc --warehouses=$wh --ramp=30m --duration=720m $COCKROACH_ADDRS
```
in COCKROACH_ADDRS I specify all CRDB instances

Note, that I don't use enterprise feature of collocation.

At some point performance degrades:
```
3595.8s 0 2659.5 2727.2 31.5 67.1 92.3 130.0 newOrder
3595.8s 0 283.6 272.7 8.4 15.2 17.8 26.2 orderStatus
3595.8s 0 2573.5 2726.6 17.8 48.2 79.7 121.6 payment
3595.8s 0 256.0 272.6 16.8 24.1 28.3 31.5 stockLevel
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
3596.8s 0 270.9 272.7 58.7 83.9 100.7 117.4 delivery
3596.8s 0 2742.8 2727.2 29.4 46.1 58.7 83.9 newOrder
3596.8s 0 252.8 272.7 8.1 12.6 15.7 22.0 orderStatus
3596.8s 0 2687.7 2726.6 16.3 29.4 37.7 71.3 payment
3596.8s 0 261.8 272.6 16.3 24.1 31.5 35.7 stockLevel
3597.8s 0 289.9 272.7 62.9 104.9 130.0 369.1 delivery
3597.8s 0 2694.4 2727.2 31.5 54.5 83.9 117.4 newOrder
3597.8s 0 245.5 272.7 8.9 17.8 30.4 32.5 orderStatus
3597.8s 0 2718.6 2726.6 16.8 31.5 52.4 335.5 payment
3597.8s 0 278.8 272.6 16.8 26.2 30.4 31.5 stockLevel
3603.9s 0 8.4 272.2 62.9 121.6 15032.4 15032.4 delivery
3603.9s 0 93.1 2723.0 33.6 79.7 15032.4 15032.4 newOrder
3603.9s 0 8.2 272.3 8.1 15.2 21.0 15032.4 orderStatus
3603.9s 0 89.8 2722.4 18.9 60.8 5637.1 15032.4 payment
3603.9s 0 9.7 272.2 17.8 32.5 15032.4 15032.4 stockLevel
3613.4s 0 8.0 271.5 15032.4 15569.3 15569.3 15569.3 delivery
3613.4s 0 120.5 2715.8 15032.4 15569.3 15569.3 15569.3 newOrder
3613.4s 0 44.2 271.6 11811.2 15032.4 15032.4 15032.4 orderStatus
3613.4s 0 144.6 2715.3 15032.4 15569.3 15569.3 15569.3 payment
3613.4s 0 56.2 271.5 11811.2 12348.0 15032.4 15032.4 stockLevel
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
3613.8s 0 0.0 271.5 0.0 0.0 0.0 0.0 delivery
3613.8s 0 32.0 2715.5 15569.3 15569.3 15569.3 15569.3 newOrder
3613.8s 0 12.8 271.5 12348.0 12348.0 12348.0 12348.0 orderStatus
3613.8s 0 51.2 2714.9 12348.0 15569.3 15569.3 15569.3 payment
3613.8s 0 19.2 271.5 12348.0 13421.8 13421.8 13421.8 stockLevel
3614.8s 0 25.6 271.4 11274.3 13958.6 16643.0 16643.0 delivery
3614.8s 0 284.5 2714.8 11274.3 16106.1 16643.0 16643.0 newOrder
3614.8s 0 237.2 271.5 10737.4 11274.3 11274.3 11811.2 orderStatus
3614.8s 0 771.7 2714.4 11274.3 12884.9 13958.6 16106.1 payment
3614.8s 0 174.2 271.4 10737.4 11274.3 11274.3 11274.3 stockLevel
3615.8s 0 84.1 271.4 11811.2 12348.0 17179.9 18253.6 delivery
3615.8s 0 1726.1 2714.7 11811.2 12348.0 13421.8 18253.6 newOrder
3615.8s 0 152.8 271.5 11274.3 12348.0 12348.0 12348.0 orderStatus
3615.8s 0 959.6 2714.0 11811.2 12348.0 12348.0 15032.4 payment
3615.8s 0 143.3 271.4 11811.2 12348.0 12348.0 12348.0 stockLevel
3616.8s 0 91.7 271.3 12348.0 12884.9 15032.4 18253.6 delivery
3616.8s 0 3171.8 2714.6 12348.0 13421.8 13421.8 18253.6 newOrder
3616.8s 0 118.5 271.5 7516.2 12884.9 12884.9 14495.5 orderStatus
3616.8s 0 1673.8 2713.6 12348.0 13421.8 13421.8 14495.5 payment
3616.8s 0 155.4 271.4 12348.0 12884.9 13421.8 14495.5 stockLevel
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
3617.8s 0 140.2 271.3 13421.8 15569.3 19327.4 19327.4 delivery
3617.8s 0 2473.2 2714.6 13421.8 14495.5 15569.3 20401.1 newOrder
3617.8s 0 90.4 271.4 4563.4 13958.6 13958.6 13958.6 orderStatus
3617.8s 0 2613.4 2713.6 13421.8 13958.6 14495.5 17179.9 payment
3617.8s 0 150.4 271.3 9126.8 13958.6 13958.6 15569.3 stockLevel
3618.8s 0 81.4 271.2 8053.1 15032.4 20401.1 20401.1 delivery
3618.8s 0 2750.7 2714.6 14495.5 15032.4 15569.3 20401.1 newOrder
3618.8s 0 139.9 271.4 5368.7 14495.5 14495.5 14495.5 orderStatus
3618.8s 0 1015.1 2713.2 14495.5 15032.4 15032.4 15569.3 payment
3618.8s 0 135.0 271.3 5368.7 14495.5 15032.4 15032.4 stockLevel
3619.8s 0 113.5 271.2 6710.9 16106.1 18253.6 21474.8 delivery
3619.8s 0 4704.6 2715.2 15569.3 16106.1 16106.1 22548.6 newOrder
3619.8s 0 174.7 271.3 6442.5 6979.3 6979.3 6979.3 orderStatus
3619.8s 0 1086.8 2712.7 6979.3 16106.1 16106.1 17179.9 payment
3619.8s 0 197.4 271.3 6442.5 6979.3 16106.1 16106.1 stockLevel
3620.8s 0 248.4 271.2 7516.2 17179.9 17179.9 19327.4 delivery
3620.8s 0 4190.7 2715.6 7784.6 17179.9 17179.9 22548.6 newOrder
3620.8s 0 258.4 271.3 7516.2 7784.6 7784.6 8053.1 orderStatus
3620.8s 0 1339.7 2712.3 7516.2 16643.0 17179.9 20401.1 payment
3620.8s 0 301.9 271.3 7516.2 7784.6 10200.5 16106.1 stockLevel
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
3621.8s 0 463.2 271.2 8321.5 9126.8 18253.6 19327.4 delivery
3621.8s 0 3801.5 2715.9 8321.5 15569.3 18253.6 19327.4 newOrder
3621.8s 0 234.6 271.3 6710.9 9126.8 9126.8 9126.8 orderStatus
3621.8s 0 2170.3 2712.2 8589.9 9126.8 18253.6 18253.6 payment
3621.8s 0 239.5 271.3 7784.6 9126.8 9126.8 17179.9 stockLevel
3622.8s 0 25.9 271.2 9126.8 9126.8 9126.8 9126.8 delivery
3622.8s 0 562.5 2715.3 9126.8 9126.8 18253.6 18253.6 newOrder
3622.8s 0 3.2 271.3 9126.8 9126.8 9126.8 9126.8 orderStatus
3622.8s 0 493.5 2711.6 9126.8 9126.8 9126.8 18253.6 payment
3622.8s 0 18.3 271.2 9126.8 9126.8 9126.8 9126.8 stockLevel
3623.8s 0 0.0 271.1 0.0 0.0 0.0 0.0 delivery
3623.8s 0 10.0 2714.5 10737.4 10737.4 10737.4 10737.4 newOrder
3623.8s 0 0.0 271.2 0.0 0.0 0.0 0.0 orderStatus
3623.8s 0 0.0 2710.8 0.0 0.0 0.0 0.0 payment
3623.8s 0 0.0 271.1 0.0 0.0 0.0 0.0 stockLevel
3624.8s 0 1.0 271.0 8321.5 8321.5 8321.5 8321.5 delivery
3624.8s 0 5.0 2713.8 11811.2 21474.8 21474.8 21474.8 newOrder
3624.8s 0 0.0 271.1 0.0 0.0 0.0 0.0 orderStatus
3624.8s 0 2.0 2710.1 9126.8 11274.3 11274.3 11274.3 payment
3624.8s 0 0.0 271.1 0.0 0.0 0.0 0.0 stockLevel
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
3625.8s 0 0.0 270.9 0.0 0.0 0.0 0.0 delivery
3625.8s 0 16.0 2713.0 12884.9 12884.9 12884.9 12884.9 newOrder
3625.8s 0 0.0 271.0 0.0 0.0 0.0 0.0 orderStatus
3625.8s 0 26.9 2709.3 12348.0 12884.9 12884.9 12884.9 payment
3625.8s 0 0.0 271.0 0.0 0.0 0.0 0.0 stockLevel
3626.8s 0 1.0 270.9 12884.9 12884.9 12884.9 12884.9 delivery
3626.8s 0 29.7 2712.3 13421.8 13958.6 23622.3 23622.3 newOrder
3626.8s 0 6.9 271.0 13958.6 23622.3 23622.3 23622.3 orderStatus
3626.8s 0 20.8 2708.6 12884.9 13958.6 16643.0 16643.0 payment
3626.8s 0 2.0 270.9 11811.2 13958.6 13958.6 13958.6 stockLevel
3627.8s 0 73.9 270.8 14495.5 24696.1 24696.1 24696.1 delivery
3627.8s 0 2140.9 2712.2 14495.5 24696.1 24696.1 25769.8 newOrder
3627.8s 0 227.5 271.0 14495.5 24696.1 24696.1 24696.1 orderStatus
3627.8s 0 1043.6 2708.1 13958.6 17179.9 24696.1 24696.1 payment
3627.8s 0 119.0 270.9 12884.9 15032.4 23622.3 24696.1 stockLevel
3628.8s 0 132.9 270.8 15569.3 24696.1 24696.1 25769.8 delivery
3628.8s 0 2796.4 2712.2 15569.3 25769.8 25769.8 30064.8 newOrder
3628.8s 0 305.1 271.0 13958.6 24696.1 24696.1 25769.8 orderStatus
3628.8s 0 787.5 2707.6 15032.4 18253.6 24696.1 25769.8 payment
3628.8s 0 199.4 270.9 13958.6 16106.1 16106.1 20401.1 stockLevel
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
3629.8s 0 230.7 270.8 15032.4 19327.4 25769.8 25769.8 delivery
3629.8s 0 2753.3 2712.2 16643.0 25769.8 26843.5 27917.3 newOrder
3629.8s 0 261.4 271.0 14495.5 16106.1 16643.0 16643.0 orderStatus
3629.8s 0 1976.1 2707.4 16643.0 25769.8 26843.5 26843.5 payment
3629.8s 0 249.5 270.8 14495.5 16106.1 16643.0 19327.4 stockLevel
3630.8s 0 166.6 270.7 15569.3 18253.6 18253.6 26843.5 delivery
3630.8s 0 3558.4 2712.4 17179.9 26843.5 26843.5 27917.3 newOrder
3630.8s 0 156.5 270.9 15032.4 18253.6 18253.6 22548.6 orderStatus
3630.8s 0 1684.3 2707.1 14495.5 26843.5 26843.5 27917.3 payment
3630.8s 0 195.9 270.8 15032.4 18253.6 18253.6 18253.6 stockLevel
3631.8s 0 300.6 270.7 16106.1 18253.6 19327.4 19327.4 delivery
3631.8s 0 3974.6 2712.8 18253.6 27917.3 27917.3 28991.0 newOrder
3631.8s 0 320.9 270.9 11811.2 16643.0 19327.4 19327.4 orderStatus
3631.8s 0 2596.6 2707.1 15032.4 18253.6 19327.4 28991.0 payment
3631.8s 0 289.4 270.8 12884.9 16643.0 18253.6 22548.6 stockLevel
3632.8s 0 23.8 270.7 20401.1 28991.0 28991.0 28991.0 delivery
3632.8s 0 25.9 2712.0 16643.0 19327.4 19327.4 19327.4 newOrder
3632.8s 0 4.1 270.9 6979.3 11274.3 11274.3 11274.3 orderStatus
3632.8s 0 36.3 2706.3 6979.3 16643.0 21474.8 21474.8 payment
3632.8s 0 45.6 270.8 20401.1 28991.0 28991.0 28991.0 stockLevel
Error: error in stockLevel: ERROR: inbox communication error: rpc error: code = Canceled desc = context canceled (SQLSTATE 58C01)
```

The same I see in the CRDB metrics (note, that before the degradation CPU utilization was just 0.75 and disk is also underutilized):


Jira issue: CRDB-36392