Sentry: conn_executor.go:1058: panic: must not be holding latches (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1 | pkg/sql/conn_ex...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7452636014/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7452636014/?referrer=webhooks_plugin)

Panic Message:

```
conn_executor.go:1058: panic: must not be holding latches
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
| pkg/sql/conn_executor.go:1058
| [...repeated from below...]
Wraps: (2) while executing: SELECT _, _ FROM _ WHERE _ IN ($1,)
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
| pkg/sql/conn_executor.go:1058
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| pkg/sql/colexecerror/error.go:137
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover
| pkg/util/stop/stopper.go:226
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1
| pkg/kv/kvserver/store_send.go:104
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard
| pkg/kv/kvserver/concurrency/concurrency_manager.go:365
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq
| pkg/kv/kvserver/concurrency/concurrency_manager.go:302
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| pkg/kv/kvserver/replica_send.go:465
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| pkg/kv/kvserver/replica_send.go:185
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| pkg/kv/kvserver/store_send.go:188
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
| pkg/kv/kvserver/stores.go:199
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| pkg/server/node.go:1665
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
| pkg/server/node.go:1858
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
| pkg/rpc/context.go:735
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:96
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
| pkg/rpc/context.go:178
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.gatewayRequestRecoveryInterceptor
| pkg/rpc/metrics.go:462
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4
| pkg/rpc/metrics.go:415
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
| pkg/rpc/context.go:136
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:349
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
| pkg/rpc/context.go:134
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
| pkg/rpc/context.go:745
| github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:236
| github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
| pkg/rpc/context.go:930
Wraps: (4) panic: must not be holding latches
| this is tracked in github.com/cockroachdb/cockroach/issues/77663; please comment if seen
| eval_kind=0, holding_latches=true, branch=1, first_iteration=true, stack=
| goroutine 271671 [running]:
| runtime/debug.Stack()
| GOROOT/src/runtime/debug/stack.go:26 +0x5e
| github.com/cockroachdb/cockroach/pkg/util/debugutil.Stack(...)
| pkg/util/debugutil/debugutil.go:63
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc004c14b90, {0x83de588, 0xc03509b3f8}, 0xc057538340, 0x1)
| pkg/kv/kvserver/concurrency/concurrency_manager.go:368 +0x105f
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc004c14b90, {0x83de588, 0xc03509b3f8}, 0x69309a0?, {0xc02f84fe60, {0x18ab397a7987a0c9, 0x0}, 0x0, 0x0, 0x0, ...}, ...)
| pkg/kv/kvserver/concurrency/concurrency_manager.go:302 +0x399
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc001b3e888, {0x83de588, 0xc03509b3f8}, 0xc02f867a40, 0x7912ee0)
| pkg/kv/kvserver/replica_send.go:465 +0x4ac
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc001b3e888, {0x83de588, 0xc03509b360}, 0xc02f867a40)
| pkg/kv/kvserver/replica_send.go:185 +0x852
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc01475f008, {0x83de588?, 0xc03509b2c8?}, 0xc02f867a40)
| pkg/kv/kvserver/store_send.go:188 +0x205c
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc000771b00, {0x83de588, 0xc03509b2c8}, 0xc02f867a40)
| pkg/kv/kvserver/stores.go:199 +0x175
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc00c52ea88, {0x83de588?, 0xc03509b2c8?}, {0x0?}, 0xc02f867a40)
| pkg/server/node.go:1665 +0x6f4
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc00c52ea88, {0x83de588, 0xc035087900}, 0xc02f867a40)
| pkg/server/node.go:1858 +0x5b5
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0x83de588?, 0xc035087900?}, {0x73eb2e0?, 0xc02f867a40?})
| pkg/rpc/context.go:735 +0x45
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x83de588, 0xc035087900}, {0x73eb2e0, 0xc02f867a40}, 0xc00f80fcc0, 0xc010d460d8)
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:96 +0x4a6
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x83de588?, 0xc035087900?}, {0x73eb2e0?, 0xc02f867a40?})
| pkg/rpc/context.go:846 +0x34
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x83de588, 0xc035087900}, {0x73eb2e0, 0xc02f867a40}, 0xd4f2940?, 0xc00f80fce0)
| pkg/rpc/context.go:178 +0x76
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x83de588?, 0xc035087900?}, {0x73eb2e0?, 0xc02f867a40?})
| pkg/rpc/context.go:846 +0x34
| github.com/cockroachdb/cockroach/pkg/rpc.gatewayRequestRecoveryInterceptor({0x83de588, 0xc035087900}, {0x73eb2e0, 0xc02f867a40}, 0x1c319d4?, 0xc00f80fd00)
| pkg/rpc/metrics.go:462 +0x244
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x83de588?, 0xc035087900?}, {0x73eb2e0?, 0xc02f867a40?})
| pkg/rpc/context.go:846 +0x34
| github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4({0x83de588, 0xc035087900}, {0x73eb2e0, 0xc02f867a40}, 0xc00f80fcc0, 0xc00f80fd20)
| pkg/rpc/metrics.go:415 +0x2b4
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x83de588?, 0xc035087900?}, {0x73eb2e0?, 0xc02f867a40?})
| pkg/rpc/context.go:846 +0x34
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x83de588?, 0xc035087900?})
| pkg/rpc/context.go:136 +0x3...
```

Stacktrace (expand for inline code snippets):

[pkg/rpc/context.go#L929-L931](pkg/rpc/context.go#L929-L931)
[pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L235-L237](pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L235-L237)
[pkg/rpc/context.go#L744-L746](pkg/rpc/context.go#L744-L746)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/context.go#L133-L135](pkg/rpc/context.go#L133-L135)
[pkg/util/stop/stopper.go#L348-L350](pkg/util/stop/stopper.go#L348-L350)
[pkg/rpc/context.go#L135-L137](pkg/rpc/context.go#L135-L137)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/metrics.go#L414-L416](pkg/rpc/metrics.go#L414-L416)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/metrics.go#L461-L463](pkg/rpc/metrics.go#L461-L463)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/context.go#L177-L179](pkg/rpc/context.go#L177-L179)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L95-L97](pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L95-L97)
[pkg/rpc/context.go#L734-L736](pkg/rpc/context.go#L734-L736)
[pkg/server/node.go#L1857-L1859](pkg/server/node.go#L1857-L1859)
[pkg/server/node.go#L1664-L1666](pkg/server/node.go#L1664-L1666)
[pkg/kv/kvserver/stores.go#L198-L200](pkg/kv/kvserver/stores.go#L198-L200)
[pkg/kv/kvserver/store_send.go#L187-L189](pkg/kv/kvserver/store_send.go#L187-L189)
[pkg/kv/kvserver/replica_send.go#L184-L186](pkg/kv/kvserver/replica_send.go#L184-L186)
[pkg/kv/kvserver/replica_send.go#L464-L466](pkg/kv/kvserver/replica_send.go#L464-L466)
[pkg/kv/kvserver/concurrency/concurrency_manager.go#L301-L303](pkg/kv/kvserver/concurrency/concurrency_manager.go#L301-L303)
[pkg/kv/kvserver/concurrency/concurrency_manager.go#L364-L366](pkg/kv/kvserver/concurrency/concurrency_manager.go#L364-L366)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)
[pkg/kv/kvserver/store_send.go#L103-L105](pkg/kv/kvserver/store_send.go#L103-L105)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)
[pkg/util/stop/stopper.go#L225-L227](pkg/util/stop/stopper.go#L225-L227)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)
[pkg/sql/colexecerror/error.go#L136-L138](pkg/sql/colexecerror/error.go#L136-L138)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)
[pkg/sql/conn_executor.go#L1057-L1059](pkg/sql/conn_executor.go#L1057-L1059)

```
pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 930
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewContext.ClientInterceptor.func8 at line 236
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 745
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 134
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 349
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 136
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/metrics.go in pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4 at line 415
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/metrics.go in pkg/rpc.gatewayRequestRecoveryInterceptor at line 462
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 178
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 96
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 735
pkg/server/node.go in pkg/server.(*Node).Batch at line 1858
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1665
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 199
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 188
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 185
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 465
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq at line 302
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard at line 365
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1 at line 104
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).recover at line 226
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 137
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn.func1 at line 1058
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v25.3.6 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.3.6 |
| Cockroach SHA | 7bad8e82ef2989b3cd657baca02567a892c1b5c3 |
| # of CPUs | 384 |
| # of Goroutines | 3169 |

Jira issue: CRDB-63461