basalt-26.3.2-rc-20260913: kv/kvserver: TestAckWriteBeforeApplication failed

kv/kvserver.TestAckWriteBeforeApplication failed on basalt-26.3.2-rc-20260913 @ c6a822fb5ec0b622c713fccacb82159d1b60bbcd:

    test_log_scope.go:82: use -show-logs to present logs inline
=== RUN   TestAckWriteBeforeApplication/numRepls=1
    test_server_shim.go:146: using test tenant configuration from testserver factory defaults
POTENTIAL DEADLOCK:
Previous place where the lock was grabbed
goroutine 1422761 lock 0x272a4119d178
pkg/kv/kvserver/replica_raft.go:879 kvserver.(*Replica).handleRaftReady ??? <<<<<
pkg/kv/kvserver/replica_raft.go:878 kvserver.(*Replica).handleRaftReady ???
pkg/kv/kvserver/store_raft.go:765 kvserver.(*Store).processReady ???
pkg/kv/kvserver/scheduler.go:422 kvserver.(*raftSchedulerShard).worker ???
pkg/kv/kvserver/scheduler.go:314 kvserver.(*raftScheduler).Start.func2 ???

Have been trying to lock it again for more than 5m0s
goroutine 5117886 lock 0x272a4119d178
pkg/kv/kvserver/replica_raftlog_writes.go:101 kvserver.(*replicaLogStorage).updateLogSize ??? <<<<<
pkg/kv/kvserver/replica_raftlog_writes.go:100 kvserver.(*replicaLogStorage).updateLogSize ???
pkg/kv/kvserver/raft_log_queue.go:676 kvserver.(*raftLogQueue).process ???
pkg/kv/kvserver/queue.go:1254 kvserver.(*baseQueue).processReplica.func1 ???
pkg/util/timeutil/timeout.go:27 timeutil.RunWithTimeout ???
pkg/kv/kvserver/queue.go:1246 kvserver.(*baseQueue).processReplica ???
pkg/kv/kvserver/queue.go:1194 kvserver.(*baseQueue).processOneAsyncAndReleaseSem.func1 ???
pkg/util/stop/stopper.go:447 stop.(*Stopper).RunAsyncTaskEx.func1 ???

Here is what goroutine 1422761 doing now
goroutine 1422761 [chan receive, 6 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestAckWriteBeforeApplication.func1.TestAckWriteBeforeApplication.func1.1.6({{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, {0x18d6f865a2874eb0, 0x0}, {0x0, ...}, ...}, ...})
	pkg/kv/kvserver/client_raft_test.go:5238 +0x3f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).prepareLocalResult(0x272a4119d008, {0x88ae498, 0x272a45962280}, 0x272a2923af10)
	pkg/kv/kvserver/replica_application_result.go:85 +0x20a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects(0x272a4119d198, {0x88ae498, 0x272a45962280}, {0x8909018?, 0x272a2923af10})
	pkg/kv/kvserver/replica_application_state_machine.go:202 +0xc5
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter({0x7f6dd3baa830, 0x272a4119d638}, 0x272a44710c18)
	pkg/kv/kvserver/apply/cmd.go:209 +0x13c
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0x272a44711438, {0x88ae498, 0x272a45962280}, {0x88ec6c0, 0x272a4119d5d8})
	pkg/kv/kvserver/apply/task.go:290 +0x1a5
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0x272a44711438, {0x88ae498, 0x272a45962280})
	pkg/kv/kvserver/apply/task.go:246 +0x92
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	pkg/kv/kvserver/replica_raft.go:1246 +0x1c85
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	pkg/kv/kvserver/replica_raft.go:881 +0x1d3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0x272a139b2008, 0x272a28f5d7a0?)
	pkg/kv/kvserver/store_raft.go:766 +0x15b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0x272a29895420, {0x88ae498, 0x272a2ed2e780}, {0x88da460, 0x272a139b2008}, 0x272a14ae6888)
	pkg/kv/kvserver/scheduler.go:421 +0x222
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x88ae498, 0x272a2ed2e780}, 0x272a142ce908?)
	pkg/kv/kvserver/scheduler.go:314 +0xe9
created by github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start in goroutine 1416962
	pkg/kv/kvserver/scheduler.go:329 +0x27e

Parameters:

• attempt=1
• deadlock=true
• run=1
• shard=14

Start by reproducing kv/kvserver.TestAckWriteBeforeApplication with attempt=1, run=1, and shard=14. Read the test at pkg/kv/kvserver/client_raft_test.go:5238 and follow the reported lock paths through replica_raft.go, replica_raftlog_writes.go, and raft_log_queue.go. Done means the test completes without the reported potential deadlock.