basalt-26.3.2-rc-20260908: kv/kvserver: TestAckWriteBeforeApplication failed

kv/kvserver.TestAckWriteBeforeApplication [failed](https://mesolite.cluster.engflow.com/invocations/default/174405c2-1c04-4c0e-af22-26f8b5cc1fe9?testReportRun=1&testReportShard=3&testReportAttempt=1#targets-Ly9wa2cva3Yva3ZzZXJ2ZXI6a3ZzZXJ2ZXJfdGVzdA==) on basalt-26.3.2-rc-20260908 @ [807558e5e81666af257c811e925e4cae570abc28](https://github.com/cockroachdb/cockroach/commits/807558e5e81666af257c811e925e4cae570abc28):

```
test_log_scope.go:82: use -show-logs to present logs inline
=== RUN TestAckWriteBeforeApplication/numRepls=1
test_server_shim.go:146: using test tenant configuration from testserver factory defaults
POTENTIAL DEADLOCK:
Previous place where the lock was grabbed
goroutine 1504553 lock 0x274173ebd178
pkg/kv/kvserver/replica_raft.go:879 kvserver.(*Replica).handleRaftReady ??? <<<<<
pkg/kv/kvserver/replica_raft.go:878 kvserver.(*Replica).handleRaftReady ???
pkg/kv/kvserver/store_raft.go:765 kvserver.(*Store).processReady ???
pkg/kv/kvserver/scheduler.go:422 kvserver.(*raftSchedulerShard).worker ???
pkg/kv/kvserver/scheduler.go:314 kvserver.(*raftScheduler).Start.func2 ???

Have been trying to lock it again for more than 5m0s
goroutine 5091601 lock 0x274173ebd178
pkg/kv/kvserver/replica_raftlog_writes.go:101 kvserver.(*replicaLogStorage).updateLogSize ??? <<<<<
pkg/kv/kvserver/replica_raftlog_writes.go:100 kvserver.(*replicaLogStorage).updateLogSize ???
pkg/kv/kvserver/raft_log_queue.go:676 kvserver.(*raftLogQueue).process ???
pkg/kv/kvserver/queue.go:1254 kvserver.(*baseQueue).processReplica.func1 ???
pkg/util/timeutil/timeout.go:27 timeutil.RunWithTimeout ???
pkg/kv/kvserver/queue.go:1246 kvserver.(*baseQueue).processReplica ???
pkg/kv/kvserver/queue.go:1194 kvserver.(*baseQueue).processOneAsyncAndReleaseSem.func1 ???
pkg/util/stop/stopper.go:447 stop.(*Stopper).RunAsyncTaskEx.func1 ???

Here is what goroutine 1504553 doing now
goroutine 1504553 [chan receive, 6 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestAckWriteBeforeApplication.func1.TestAckWriteBeforeApplication.func1.1.6({{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, {0x18d51fdf7f36b8f2, 0x0}, {0x0, ...}, ...}, ...})
pkg/kv/kvserver/client_raft_test.go:5238 +0x3f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).prepareLocalResult(0x274173ebd008, {0x882cc98, 0x274170bb12c0}, 0x274151fa2a10)
pkg/kv/kvserver/replica_application_result.go:85 +0x20a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects(0x274173ebd198, {0x882cc98, 0x274170bb12c0}, {0x8887778?, 0x274151fa2a10})
pkg/kv/kvserver/replica_application_state_machine.go:202 +0xc5
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter({0x7fbf39ec5318, 0x274173ebd638}, 0x2741715bec18)
pkg/kv/kvserver/apply/cmd.go:209 +0x13c
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0x2741715bf438, {0x882cc98, 0x274170bb12c0}, {0x886aee0, 0x274173ebd5d8})
pkg/kv/kvserver/apply/task.go:290 +0x1a5
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0x2741715bf438, {0x882cc98, 0x274170bb12c0})
pkg/kv/kvserver/apply/task.go:246 +0x92
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
pkg/kv/kvserver/replica_raft.go:1246 +0x1c85
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
pkg/kv/kvserver/replica_raft.go:881 +0x1d3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0x274140a7c008, 0x1fe?)
pkg/kv/kvserver/store_raft.go:766 +0x15b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0x2741548fe1c0, {0x882cc98, 0x2741658c4320}, {0x8858c60, 0x274140a7c008}, 0x27413e6aa888)
pkg/kv/kvserver/scheduler.go:421 +0x222
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x882cc98, 0x2741658c4320}, 0x274162fa7c08?)
pkg/kv/kvserver/scheduler.go:314 +0xe9
created by github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start in goroutine 1498829
pkg/kv/kvserver/scheduler.go:329 +0x27e

```

Parameters:
- attempt=1
- deadlock=true
- run=1
- shard=3
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #172478 kv/kvserver: TestAckWriteBeforeApplication failed [A-testing C-bug C-test-failure O-robot P-3 T-kv branch-release-25.4]

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestAckWriteBeforeApplication.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)