basalt-26.3.2-rc-20260914: testutils/testcluster: TestStopServer failed

testutils/testcluster.TestStopServer [failed](https://mesolite.cluster.engflow.com/invocations/default/e97a94d3-9833-4a85-9ee3-760c3aa12029?testReportRun=3&testReportShard=2&testReportAttempt=1#targets-Ly9wa2cvdGVzdHV0aWxzL3Rlc3RjbHVzdGVyOnRlc3RjbHVzdGVyX3Rlc3Q=) on basalt-26.3.2-rc-20260914 @ [d3eb08ebdd1dacc8d3dac0b107a7db5929e09969](https://github.com/cockroachdb/cockroach/commits/d3eb08ebdd1dacc8d3dac0b107a7db5929e09969):

```
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc01ff79420)
external/org_golang_google_grpc/internal/transport/controlbuf.go:552 +0x153
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
external/org_golang_google_grpc/internal/transport/http2_client.go:451 +0x156
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 419212
external/org_golang_google_grpc/internal/transport/http2_client.go:449 +0x388b

goroutine 421143 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).processQueue(0xc003b2cb60, {0xa3d6a30, 0xc0267b51a0}, 0xc01d4a0f78, {0xa3b1840, 0xc01cf36aa8}, 0xa345498?)
pkg/kv/kvserver/raft_transport.go:693 +0x9e5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).startProcessNewQueue.func2({0xa3d6a30, 0xc0267b51a0})
pkg/kv/kvserver/raft_transport.go:919 +0x35e
github.com/cockroachdb/cockroach/pkg/util/pprofutil.Do({0xa3d8518, 0xc028f732c0}, 0xc00418ec20, {0xc02b6e5f58, 0x4, 0x4})
pkg/util/pprofutil/labels.go:38 +0xa3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).startProcessNewQueue.func3({0xa3d8518, 0xc028f732c0})
pkg/kv/kvserver/raft_transport.go:935 +0x22c
created by github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).startProcessNewQueue in goroutine 421072
pkg/kv/kvserver/raft_transport.go:933 +0x565

goroutine 414676 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider.func1()
pkg/util/admission/store_grant_coordinator.go:158 +0x196
created by github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider in goroutine 413084
pkg/util/admission/store_grant_coordinator.go:148 +0x6ff

goroutine 419442 [select, 4 minutes]:
google.golang.org/grpc.newClientStreamWithParams.func4()
external/org_golang_google_grpc/stream.go:383 +0xe7
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 419190
external/org_golang_google_grpc/stream.go:382 +0x1c9f

goroutine 413139 [select, 4 minutes]:
github.com/cockroachdb/pebble.(*ConcurrencyLimitScheduler).granter(0xc02999fd10)
external/com_github_cockroachdb_pebble/compaction_scheduler.go:470 +0x17a
created by github.com/cockroachdb/pebble.(*ConcurrencyLimitScheduler).Register in goroutine 413084
external/com_github_cockroachdb_pebble/compaction_scheduler.go:359 +0x185

goroutine 418823 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc01ebb5068, 0x418)
GOROOT/src/runtime/sema.go:617 +0x1b3
sync.(*Cond).Wait(0xc01ebb5058)
GOROOT/src/sync/cond.go:71 +0x55
github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*updatesBuf).GetBySeq(0xc01ebb5040, {0xa3d69f8, 0xc003714ff0}, 0x431)
pkg/kv/kvserver/closedts/sidetransport/sender.go:759 +0x1fa
github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*rpcConn).run.func1({0xa3d8518, 0xc01946a280})
pkg/kv/kvserver/closedts/sidetransport/sender.go:983 +0x3ff
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0xa3d8518, 0xc01946a280})
pkg/util/stop/stopper.go:447 +0x113
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 417736
pkg/util/stop/stopper.go:445 +0x247
```

Parameters:
- attempt=1
- race=true
- run=3
- shard=2
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestStopServer.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

The reported entry point is testutils/testcluster.TestStopServer; begin by rerunning it with race=true, run=3, and shard=2 and inspect the full failure output. Trace the server-stop path through the files named in the stack trace, especially pkg/kv/kvserver/raft_transport.go and pkg/kv/kvserver/closedts/sidetransport/sender.go. Done means identifying the cause and making the test pass without a race failure.

Written by the indexing model from the issue text.