release-26.2: kv/kvserver: TestBackpressureNotAppliedWhenReducingRangeSize failed [port reuse]

kv/kvserver.TestBackpressureNotAppliedWhenReducingRangeSize [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/21321543?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/21321543?buildTab=artifacts#/) on release-26.2 @ [0b020fad09d2237bbec5a91741293ea0fdf0c8fe](https://github.com/cockroachdb/cockroach/commits/0b020fad09d2237bbec5a91741293ea0fdf0c8fe):

Failed with:

```
=== RUN TestBackpressureNotAppliedWhenReducingRangeSize
test_log_scope.go:172: test logs captured to: /artifacts/tmp/_tmp/ac3e4004160555c7c575e1264d1b8f5f/logTestBackpressureNotAppliedWhenReducingRangeSize3742233884
test_log_scope.go:82: use -show-logs to present logs inline
=== RUN TestBackpressureNotAppliedWhenReducingRangeSize/no_backpressure_when_much_larger_on_new_node
test_server_shim.go:146: using test tenant configuration from testserver factory defaults
test_server_shim.go:638: DRPC is enabled (via testserver factory defaults)
```
Fatal entries found in Cockroach logs:

```
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 match(19) is out of range [lastIndex(0)]. Was the raft log corrupted, truncated, or lost?
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !goroutine 53123 [running]:
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !runtime/debug.Stack()
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! GOROOT/src/runtime/debug/stack.go:26 +0x64
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/util/debugutil.Stack(...)
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/util/debugutil/debugutil.go:63
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x401cd3e5a0, 0x24}, {0x6e1d0ca, 0x1}, {0x6e1d0c9, 0x1}, {0x6d3c86d, 0x6}, {0x6e1d0ca, ...}}, ...})
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/util/log/clog.go:292 +0xc4
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x7e24978, 0x40126bc460}, 0x3, 0x4, 0xf, 0x0?, {0x6fd7dd9, 0x5a}, {0x4004a96ca0, 0x2, ...})
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/util/log/channels.go:104 +0x444
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/util/log/channels.go:34
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/util/log.loggerKvExec.FatalfDepth(...)
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! bazel-out/aarch64-fastbuild/bin/pkg/util/log/log_channels_generated.go:6624
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftLogger).Panicf(0x401c5f1950, {0x6fd7dd9, 0x5a}, {0x4004a96ca0, 0x2, 0x2})
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/kv/kvserver/raft.go:114 +0x94
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/raft.(*raft).checkMatch(0x401d770820, 0x13)
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/raft/raft.go:2422 +0xf4
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/raft.(*raft).handleAppendEntries(0x401d770820, {0x3, 0x2, 0x1, 0x6, 0x6, 0x15, {0x4016d64870, 0x1, 0x1}, ...})
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/raft/raft.go:2336 +0x2c
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/raft.stepFollower(0x401d770820, {0x3, 0x2, 0x1, 0x6, 0x6, 0x15, {0x4016d64870, 0x1, 0x1}, ...})
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/raft/raft.go:2236 +0x360
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/raft.(*raft).Step(0x401d770820, {0x3, 0x2, 0x1, 0x6, 0x6, 0x15, {0x4016d64870, 0x1, 0x1}, ...})
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/raft/raft.go:1760 +0xe28
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/raft.(*RawNode).Step(0x0?, {0x3, 0x2, 0x1, 0x6, 0x6, 0x15, {0x4016d64870, 0x1, 0x1}, ...})
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/raft/rawnode.go:144 +0xe0
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).stepRaftGroupRaftMuLocked.func1(0x4012457840)
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/kv/kvserver/replica_raft.go:746 +0x370
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked(0x40074b0008, 0x290e910?)
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/kv/kvserver/replica_raft.go:2266 +0x3c
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroup(0x40074b0008, 0x4015ad9a80)
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/kv/kvserver/replica_raft.go:2303 +0x6c
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).stepRaftGroupRaftMuLocked(0x40074b0008, 0x400f5202c0)
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/kv/kvserver/replica_raft.go:673 +0x64
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftRequestWithReplica(0x0?, {0x7e24978, 0x40126bc5a0}, 0x40074b0008, 0x400f5202c0)
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/kv/kvserver/store_raft.go:479 +0x28c
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRequestQueue.func1({0x400fd5fc08?, 0x7e24978?}, 0x4010faa960?)
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/kv/kvserver/store_raft.go:705 +0x30
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest(0x400fd606c8?, {0x7e24978, 0x4010faa960}, 0x400f5202c0, 0x4015ad9e58)
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/kv/kvserver/store_raft.go:434 +0xb4
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRequestQueue(0x400fd5fc08, {0x7e24978, 0x4010faa960}, 0x53)
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/kv/kvserver/store_raft.go:703 +0x140
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0x40246f8bd0, {0x7e24978, 0x4010faa960}, {0x7e4f060, 0x400fd5fc08}, 0x4006af3008)
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/kv/kvserver/scheduler.go:399 +0x1c4
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x7e24978, 0x4010faa960}, 0x7e24978?)
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/kv/kvserver/scheduler.go:314 +0xcc
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !created by github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start in goroutine 52980
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! pkg/kv/kvserver/scheduler.go:329 +0x224
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !For more context, check log files in: /artifacts/tmp/_tmp/ac3e4004160555c7c575e1264d1b8f5f/logTestBackpressureNotAppliedWhenReducingRangeSize3742233884
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !****************************************************************************
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !This node experienced a fatal error (printed above), and as a result the
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !process is terminating.
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !will try to determine the root cause, recommend next steps, and we can
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !improve CockroachDB based on your report.
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !Please submit a crash report by following the instructions here:
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! https://github.com/cockroachdb/cockroach/issues/new/choose
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !If you would rather not post publicly, please contact us directly at:
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 ! support@cockroachlabs.com
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !
F260421 17:59:05.610437 53123 15@raft/raft.go:2422 [T1,Vsystem,n2,s2,r83/2:{-}] 1 !The Cockroach Labs team appreciates your feedback.
```
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachdb/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestBackpressureNotAppliedWhenReducingRangeSize.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-63126

Epic CRDB-55382