Sentry: panic.go:770: panic: must not be holding latches (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:770 | github.com/cockroachdb/cockroach/pkg/kv...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7326195923/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7326195923/?referrer=webhooks_plugin)

Panic Message:

```
panic.go:770: panic: must not be holding latches
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:770
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:109
| runtime.gopanic
| GOROOT/src/runtime/panic.go:770
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:307
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:244
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:182
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| github.com/cockroachdb/cockroach/pkg/server/node.go:1436
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
| github.com/cockroachdb/cockroach/pkg/server/node.go:1631
| github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10632
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
| google.golang.org/grpc.getChainUnaryHandler.func1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169
| google.golang.org/grpc.getChainUnaryHandler.func1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163
| github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95
| google.golang.org/grpc.getChainUnaryHandler.func1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
| google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154
| github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler
| github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10634
| google.golang.org/grpc.(*Server).processUnaryRPC
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336
| google.golang.org/grpc.(*Server).handleStream
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704
| google.golang.org/grpc.(*Server).serveStreams.func1.2
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:965
| runtime.goexit
| src/runtime/asm_arm64.s:1222
Wraps: (2) panic: must not be holding latches
| this is tracked in github.com/cockroachdb/cockroach/issues/77663; please comment if seen
| eval_kind=0, holding_latches=true, branch=1, first_iteration=true, stack=
| goroutine 20778 [running]:
| runtime/debug.Stack()
| GOROOT/src/runtime/debug/stack.go:24 +0x64
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0x4001b610e0, {0x75d8b30, 0x407c8c7d10}, 0x400fe789c0, 0x1)
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:310 +0x854
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0x4001b610e0, {0x75d8b30, 0x407c8c7d10}, 0x407d57da58?, {0x407d54b8c0, {0x189baeb43983885e, 0x0}, 0x0, 0x0, 0x0, ...}, ...)
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:244 +0x228
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0x4002e4e008, {0x75d8b30, 0x407c8c7d10}, 0x407d54b560, 0x61bdb18)
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x1fc
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0x4002e4e008, {0x75d8b30?, 0x407c8c7ce0?}, 0x407d54b560)
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:182 +0x49c
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0x40016bca88, {0x75d8b30?, 0x407c8c7cb0?}, 0x407d54b560)
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x5a4
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0x4001e14000, {0x75d8b30, 0x407c8c7cb0}, 0x407d54b560)
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203 +0x60
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0x4001559c08, {0x75d8b30?, 0x407c8c7c80?}, {0x40041049c0?}, 0x407d54b560)
| github.com/cockroachdb/cockroach/pkg/server/node.go:1436 +0x3f0
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0x4001559c08, {0x75d8b30, 0x407c8c7bf0}, 0x407d54b560)
| github.com/cockroachdb/cockroach/pkg/server/node.go:1631 +0x234
| github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1({0x75d8b30?, 0x407c8c7bf0?}, {0x5cde820?, 0x407d54b560?})
| github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10632 +0xd0
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x75d8b30, 0x407c8c7bf0}, {0x5cde820, 0x407d54b560}, 0x407d52f1c0, 0x407c855818)
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x32c
| google.golang.org/grpc.getChainUnaryHandler.func1({0x75d8b30, 0x407c8c7bf0}, {0x5cde820, 0x407d54b560})
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xa0
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x75d8b30, 0x407c8c7bf0}, {0x5cde820, 0x407d54b560}, 0x407d52f1c0?, 0x4078a8db80)
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0x88
| google.golang.org/grpc.getChainUnaryHandler.func1({0x75d8b30, 0x407c8c7bf0}, {0x5cde820, 0x407d54b560})
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xa0
| github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0x4002434000?, {{0x5935de0?}, {0x76235d8?, 0x4003bf4780?}}}, {0x75d8b30, 0x407c8c7bf0}, {0x5cde820, 0x407d54b560}, 0x407d52f1c0, 0x4078a8db00)
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95 +0x15c
| google.golang.org/grpc.getChainUnaryHandler.func1({0x75d8b30, 0x407c8c7bf0}, {0x5cde820, 0x407d54b560})
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xa0
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x75d8b30?, 0x407c8c7bf0?})
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x40
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x4003822120, {0...
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1221-L1223](src/runtime/asm_arm64.s#L1221-L1223)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L964-L966](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L964-L966)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1703-L1705](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1703-L1705)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1335-L1337](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1335-L1337)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go#L10633-L10635](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go#L10633-L10635)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1153-L1155](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1153-L1155)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L133-L135](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L133-L135)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L336-L338](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L336-L338)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L135-L137](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L135-L137)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/auth.go#L94-L96](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/auth.go#L94-L96)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L168-L170](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/rpc/pkg/rpc/context.go#L168-L170)
[google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L96-L98](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L96-L98)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go#L10631-L10633](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go#L10631-L10633)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/server/node.go#L1630-L1632](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/server/node.go#L1630-L1632)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/server/node.go#L1435-L1437](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/server/node.go#L1435-L1437)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L202-L204](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L202-L204)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L192-L194](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L192-L194)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L181-L183](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L181-L183)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L467-L469](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L467-L469)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/concurrency/concurrency_manager.go#L243-L245](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/concurrency/concurrency_manager.go#L243-L245)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/concurrency/concurrency_manager.go#L306-L308](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/concurrency/concurrency_manager.go#L306-L308)
[GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L108-L110](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L108-L110)
[GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1222
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.2 at line 965
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1704
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processUnaryRPC at line 1336
pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb._Internal_Batch_Handler at line 10634
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1 at line 1154
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 134
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 337
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 136
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1163
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 95
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1163
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 169
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1163
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 97
pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb._Internal_Batch_Handler.func1 at line 10632
pkg/server/node.go in pkg/server.(*Node).Batch at line 1631
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1436
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 193
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 182
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 468
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq at line 244
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard at line 307
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1 at line 109
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.1.2 |
| Go Version | go1.22.3 X:nocoverageredesign |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v24.1.2 |
| Cockroach SHA | 7e81be6de75205c3d08b0d8dcc6ca188306abc27 |
| # of CPUs | 12 |
| # of Goroutines | 638 |

Jira issue: CRDB-61364