Sentry: panic.go:791: panic: must not be holding latches (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:791 | github.com/cockroachdb/cockroach/pkg/kv...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7372159152/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7372159152/?referrer=webhooks_plugin)

Panic Message:

```
panic.go:791: panic: must not be holding latches
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1
| pkg/kv/kvserver/store_send.go:104
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard
| pkg/kv/kvserver/concurrency/concurrency_manager.go:365
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq
| pkg/kv/kvserver/concurrency/concurrency_manager.go:302
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| pkg/kv/kvserver/replica_send.go:465
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| pkg/kv/kvserver/replica_send.go:189
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| pkg/kv/kvserver/store_send.go:188
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
| pkg/kv/kvserver/stores.go:199
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| pkg/server/node.go:1665
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
| pkg/server/node.go:1858
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
| pkg/rpc/context.go:735
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:96
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
| pkg/rpc/context.go:178
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.gatewayRequestRecoveryInterceptor
| pkg/rpc/metrics.go:462
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4
| pkg/rpc/metrics.go:415
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
| pkg/rpc/context.go:136
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:349
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
| pkg/rpc/context.go:134
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
| pkg/rpc/context.go:745
| github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:236
| github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
| pkg/rpc/context.go:930
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
| pkg/rpc/context.go:815
| github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
| pkg/rpc/context.go:938
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
| pkg/kv/kvclient/kvcoord/transport.go:211
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
| pkg/kv/kvclient/kvcoord/transport.go:189
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
| pkg/kv/kvclient/kvcoord/dist_sender.go:2780
Wraps: (2) panic: must not be holding latches
| this is tracked in github.com/cockroachdb/cockroach/issues/77663; please comment if seen
| eval_kind=0, holding_latches=true, branch=1, first_iteration=true, stack=
| goroutine 302825 [running]:
| runtime/debug.Stack()
| GOROOT/src/runtime/debug/stack.go:26 +0x5e
| github.com/cockroachdb/cockroach/pkg/util/debugutil.Stack(...)
| pkg/util/debugutil/debugutil.go:63
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc03f4483c0, {0x83de588, 0xc044ea8780}, 0xc0405b4000, 0x1)
| pkg/kv/kvserver/concurrency/concurrency_manager.go:368 +0x105f
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc03f4483c0, {0x83de588, 0xc044ea8780}, 0x2?, {0xc047a6cc60, {0x18a146f7ff03c3bf, 0x0}, 0x0, 0x0, 0x0, ...}, ...)
| pkg/kv/kvserver/concurrency/concurrency_manager.go:302 +0x399
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc0029b6d88, {0x83de588, 0xc044ea8780}, 0xc06f008280, 0x7912ee8)
| pkg/kv/kvserver/replica_send.go:465 +0x4ac
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc0029b6d88, {0x83de588, 0xc02fddbb78}, 0xc06f008140)
| pkg/kv/kvserver/replica_send.go:189 +0x8b6
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc012499808, {0x83de588?, 0xc02fddbae0?}, 0xc06f008140)
| pkg/kv/kvserver/store_send.go:188 +0x205c
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc001af9180, {0x83de588, 0xc02fddbae0}, 0xc06f008140)
| pkg/kv/kvserver/stores.go:199 +0x175
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc004b69508, {0x83de588?, 0xc02fddba48?}, {0x0?}, 0xc06f008140)
| pkg/server/node.go:1665 +0x6f4
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc004b69508, {0x83de588, 0xc044ea86e0}, 0xc06f008140)
| pkg/server/node.go:1858 +0x5b5
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0x83de588?, 0xc044ea86e0?}, {0x73eb2e0?, 0xc06f008140?})
| pkg/rpc/context.go:735 +0x45
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x83de588, 0xc044ea86e0}, {0x73eb2e0, 0xc06f008140}, 0xc00f0c1540, 0xc00ebaad50)
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:96 +0x4a6
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x83de588?, 0xc044ea86e0?}, {0x73eb2e0?, 0xc06f008140?})
| pkg/rpc/context.go:846 +0x34
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x83de588, 0xc044ea86e0}, {0x73eb2e0, 0xc06f008140}, 0xd4f2940?, 0xc00f0c1560)
| pkg/rpc/context.go:178 +0x76
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x83de588?, 0xc044ea86e0?}, {0x73eb2e0?, 0xc06f008140?})
| pkg/rpc/context.go:846 +0x34
| github.com/cockroachdb/cockroach/pkg/rpc.gatewayRequestRecoveryInterceptor({0x83de588, 0xc044ea86e0}, {0x73eb2e0, 0xc06f008140}, 0x0?, 0xc00f0c1580)
| pkg/rpc/metrics.go:462 +0x244
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x83de588?, 0xc044ea86e0?}, {0x73eb2e0?, 0xc06f008140?})
| pkg/rpc/context.go:846 +0x34
| github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4({0x83de588, 0xc044ea86e0}, {0x73eb2e0, 0xc06f008140}, 0xc00f0c1540, 0xc00f0c15a0)
| pkg/rpc/metrics.go:415 +0x2b4
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x83de588?, 0xc044ea86e0?}, {0x73eb2e0?, 0xc06f008140?})
| pkg/rpc/context.go:846 +0x34
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x83de588?, 0xc044ea86e0?})
| pkg/rpc/context.go:136 +0x36
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc004b66120, {0x83de588, 0xc044ea86e...
```

Stacktrace (expand for inline code snippets):

[pkg/kv/kvclient/kvcoord/dist_sender.go#L2779-L2781](pkg/kv/kvclient/kvcoord/dist_sender.go#L2779-L2781)
[pkg/kv/kvclient/kvcoord/transport.go#L188-L190](pkg/kv/kvclient/kvcoord/transport.go#L188-L190)
[pkg/kv/kvclient/kvcoord/transport.go#L210-L212](pkg/kv/kvclient/kvcoord/transport.go#L210-L212)
[pkg/rpc/context.go#L937-L939](pkg/rpc/context.go#L937-L939)
[pkg/rpc/context.go#L814-L816](pkg/rpc/context.go#L814-L816)
[pkg/rpc/context.go#L929-L931](pkg/rpc/context.go#L929-L931)
[pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L235-L237](pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L235-L237)
[pkg/rpc/context.go#L744-L746](pkg/rpc/context.go#L744-L746)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/context.go#L133-L135](pkg/rpc/context.go#L133-L135)
[pkg/util/stop/stopper.go#L348-L350](pkg/util/stop/stopper.go#L348-L350)
[pkg/rpc/context.go#L135-L137](pkg/rpc/context.go#L135-L137)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/metrics.go#L414-L416](pkg/rpc/metrics.go#L414-L416)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/metrics.go#L461-L463](pkg/rpc/metrics.go#L461-L463)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/context.go#L177-L179](pkg/rpc/context.go#L177-L179)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L95-L97](pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L95-L97)
[pkg/rpc/context.go#L734-L736](pkg/rpc/context.go#L734-L736)
[pkg/server/node.go#L1857-L1859](pkg/server/node.go#L1857-L1859)
[pkg/server/node.go#L1664-L1666](pkg/server/node.go#L1664-L1666)
[pkg/kv/kvserver/stores.go#L198-L200](pkg/kv/kvserver/stores.go#L198-L200)
[pkg/kv/kvserver/store_send.go#L187-L189](pkg/kv/kvserver/store_send.go#L187-L189)
[pkg/kv/kvserver/replica_send.go#L188-L190](pkg/kv/kvserver/replica_send.go#L188-L190)
[pkg/kv/kvserver/replica_send.go#L464-L466](pkg/kv/kvserver/replica_send.go#L464-L466)
[pkg/kv/kvserver/concurrency/concurrency_manager.go#L301-L303](pkg/kv/kvserver/concurrency/concurrency_manager.go#L301-L303)
[pkg/kv/kvserver/concurrency/concurrency_manager.go#L364-L366](pkg/kv/kvserver/concurrency/concurrency_manager.go#L364-L366)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)
[pkg/kv/kvserver/store_send.go#L103-L105](pkg/kv/kvserver/store_send.go#L103-L105)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)

```
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2780
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 189
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 211
pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 938
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 815
pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 930
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewContext.ClientInterceptor.func8 at line 236
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 745
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 134
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 349
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 136
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/metrics.go in pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4 at line 415
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/metrics.go in pkg/rpc.gatewayRequestRecoveryInterceptor at line 462
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 178
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 96
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 735
pkg/server/node.go in pkg/server.(*Node).Batch at line 1858
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1665
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 199
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 188
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 189
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 465
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq at line 302
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard at line 365
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1 at line 104
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v25.3.6 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.3.6 |
| Cockroach SHA | 7bad8e82ef2989b3cd657baca02567a892c1b5c3 |
| # of CPUs | 192 |
| # of Goroutines | 3169 |

Jira issue: CRDB-62195