Sentry: panic.go:770: panic: must not be holding latches (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:770 | github.com/cockroachdb/cockroach/pkg/kv...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7256822277/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7256822277/?referrer=webhooks_plugin)

Panic Message:

```
panic.go:770: panic: must not be holding latches
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:770
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1
| pkg/kv/kvserver/store_send.go:104
| runtime.gopanic
| GOROOT/src/runtime/panic.go:770
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard
| pkg/kv/kvserver/concurrency/concurrency_manager.go:302
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq
| pkg/kv/kvserver/concurrency/concurrency_manager.go:239
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| pkg/kv/kvserver/replica_send.go:453
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| pkg/kv/kvserver/replica_send.go:173
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| pkg/kv/kvserver/store_send.go:188
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
| pkg/kv/kvserver/stores.go:197
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| pkg/server/node.go:1636
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
| pkg/server/node.go:1837
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
| pkg/rpc/context.go:713
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:92
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:824
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
| pkg/rpc/context.go:165
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:824
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
| pkg/rpc/context.go:132
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:349
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
| pkg/rpc/context.go:130
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:824
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
| pkg/rpc/context.go:723
| github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:223
| github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
| pkg/rpc/context.go:908
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
| pkg/rpc/context.go:793
| github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
| pkg/rpc/context.go:916
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
| pkg/kv/kvclient/kvcoord/transport.go:209
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
| pkg/kv/kvclient/kvcoord/transport.go:187
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
| pkg/kv/kvclient/kvcoord/dist_sender.go:2747
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
| pkg/kv/kvclient/kvcoord/dist_sender.go:2202
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatchAsync.func1
| pkg/kv/kvclient/kvcoord/dist_sender.go:2062
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:498
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (2) panic: must not be holding latches
| this is tracked in github.com/cockroachdb/cockroach/issues/77663; please comment if seen
| eval_kind=0, holding_latches=true, branch=1, first_iteration=true, stack=
| goroutine 378768 [running]:
| runtime/debug.Stack()
| GOROOT/src/runtime/debug/stack.go:24 +0x5e
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc00d1f93b0, {0x867d4a0, 0xc05e0cc060}, 0xc07ee59ce0, 0x1)
| pkg/kv/kvserver/concurrency/concurrency_manager.go:305 +0x9f5
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc00d1f93b0, {0x867d4a0, 0xc05e0cc060}, 0xc05e0cc030?, {0xc04a07a240, {0x1893066fdc873360, 0x0}, 0x0, 0x0, 0x0, ...}, ...)
| pkg/kv/kvserver/concurrency/concurrency_manager.go:239 +0x29b
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00e220d88, {0x867d4a0, 0xc05e0cc060}, 0xc046499680, 0x723d240)
| pkg/kv/kvserver/replica_send.go:453 +0x34d
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc00e220d88, {0x867d4a0?, 0xc09ecf7f50?}, 0xc046499680)
| pkg/kv/kvserver/replica_send.go:173 +0x66f
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc00d9f4008, {0x867d4a0?, 0xc09ecf7f20?}, 0xc046499680)
| pkg/kv/kvserver/store_send.go:188 +0x845
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc00173f480, {0x867d4a0, 0xc09ecf7f20}, 0xc046499680)
| pkg/kv/kvserver/stores.go:197 +0x7e
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc009996a88, {0x867d4a0?, 0xc09ecf7ef0?}, {0x4?}, 0xc046499680)
| pkg/server/node.go:1636 +0x579
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc009996a88, {0x867d4a0, 0xc09ecf7e30}, 0xc046499680)
| pkg/server/node.go:1837 +0x272
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0x867d4a0?, 0xc09ecf7e30?}, {0x6d01080?, 0xc046499680?})
| pkg/rpc/context.go:713 +0x45
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x867d4a0, 0xc09ecf7e30}, {0x6d01080, 0xc046499680}, 0xc00d017ea0, 0xc00d0b3ad0)
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:92 +0x4b2
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x867d4a0?, 0xc09ecf7e30?}, {0x6d01080?, 0xc046499680?})
| pkg/rpc/context.go:824 +0x34
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x867d4a0, 0xc09ecf7e30}, {0x6d01080, 0xc046499680}, 0xc028e527c0?, 0xc00d017ec0)
| pkg/rpc/context.go:165 +0x76
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x867d4a0?, 0xc09ecf7e30?}, {0x6d01080?, 0xc046499680?})
| pkg/rpc/context.go:824 +0x34
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x867d4a0?, 0xc09ecf7e30?})
| pkg/rpc/context.go:132 +0x36
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0097122d0, {0x867d4a0, 0xc09ecf7e30}, {0x6e6e57d, 0x21}, 0xc028e528c0)
| pkg/util/stop/stopper.go:349 +0x15d
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x867d4a0?, 0xc09ecf7e30?}, {0x6d01080?, 0xc046499680?}, 0xc028e52948?, 0x50b045?)
| pkg/rpc/context.go:130 +0x8f
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x867d4a0?, 0xc09ecf7e30?}, {0x6d01080?, 0xc046499680?})
| pkg/rpc/context.go:824 +0x34
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2({0x867d4a0?, 0xc09ecf7e30?}, {0x2a?, 0x30?}, {0x6d01080?, 0xc046499680?}, {0x6b65660, 0xc0c630c600}, 0xc09ecf7e30?, {0x0, ...})
| pkg/rpc/context.go:723 +0x4c
| github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8({0x867d4a0, 0xc09ecf7e30}, {0x6e6e57d, 0x21}, {0x6d01080, 0xc046499680}, {0x6b65660, 0xc0c630c600}, 0x0, 0xc00d22c7d0, ...)
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:223 +0x463
| github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func...
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[pkg/util/stop/stopper.go#L497-L499](pkg/util/stop/stopper.go#L497-L499)
[pkg/kv/kvclient/kvcoord/dist_sender.go#L2061-L2063](pkg/kv/kvclient/kvcoord/dist_sender.go#L2061-L2063)
[pkg/kv/kvclient/kvcoord/dist_sender.go#L2201-L2203](pkg/kv/kvclient/kvcoord/dist_sender.go#L2201-L2203)
[pkg/kv/kvclient/kvcoord/dist_sender.go#L2746-L2748](pkg/kv/kvclient/kvcoord/dist_sender.go#L2746-L2748)
[pkg/kv/kvclient/kvcoord/transport.go#L186-L188](pkg/kv/kvclient/kvcoord/transport.go#L186-L188)
[pkg/kv/kvclient/kvcoord/transport.go#L208-L210](pkg/kv/kvclient/kvcoord/transport.go#L208-L210)
[pkg/rpc/context.go#L915-L917](pkg/rpc/context.go#L915-L917)
[pkg/rpc/context.go#L792-L794](pkg/rpc/context.go#L792-L794)
[pkg/rpc/context.go#L907-L909](pkg/rpc/context.go#L907-L909)
[pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L222-L224](pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L222-L224)
[pkg/rpc/context.go#L722-L724](pkg/rpc/context.go#L722-L724)
[pkg/rpc/context.go#L823-L825](pkg/rpc/context.go#L823-L825)
[pkg/rpc/context.go#L129-L131](pkg/rpc/context.go#L129-L131)
[pkg/util/stop/stopper.go#L348-L350](pkg/util/stop/stopper.go#L348-L350)
[pkg/rpc/context.go#L131-L133](pkg/rpc/context.go#L131-L133)
[pkg/rpc/context.go#L823-L825](pkg/rpc/context.go#L823-L825)
[pkg/rpc/context.go#L164-L166](pkg/rpc/context.go#L164-L166)
[pkg/rpc/context.go#L823-L825](pkg/rpc/context.go#L823-L825)
[pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L91-L93](pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L91-L93)
[pkg/rpc/context.go#L712-L714](pkg/rpc/context.go#L712-L714)
[pkg/server/node.go#L1836-L1838](pkg/server/node.go#L1836-L1838)
[pkg/server/node.go#L1635-L1637](pkg/server/node.go#L1635-L1637)
[pkg/kv/kvserver/stores.go#L196-L198](pkg/kv/kvserver/stores.go#L196-L198)
[pkg/kv/kvserver/store_send.go#L187-L189](pkg/kv/kvserver/store_send.go#L187-L189)
[pkg/kv/kvserver/replica_send.go#L172-L174](pkg/kv/kvserver/replica_send.go#L172-L174)
[pkg/kv/kvserver/replica_send.go#L452-L454](pkg/kv/kvserver/replica_send.go#L452-L454)
[pkg/kv/kvserver/concurrency/concurrency_manager.go#L238-L240](pkg/kv/kvserver/concurrency/concurrency_manager.go#L238-L240)
[pkg/kv/kvserver/concurrency/concurrency_manager.go#L301-L303](pkg/kv/kvserver/concurrency/concurrency_manager.go#L301-L303)
[GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771)
[pkg/kv/kvserver/store_send.go#L103-L105](pkg/kv/kvserver/store_send.go#L103-L105)
[GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 498
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatchAsync.func1 at line 2062
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 2202
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2747
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 187
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 209
pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 916
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 793
pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 908
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewContext.ClientInterceptor.func8 at line 223
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 723
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 824
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 130
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 349
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 132
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 824
pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 165
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 824
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 92
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 713
pkg/server/node.go in pkg/server.(*Node).Batch at line 1837
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1636
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 197
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 188
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 173
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 453
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq at line 239
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard at line 302
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1 at line 104
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v24.3.3 |
| Go Version | go1.22.8 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.3.3 |
| Cockroach SHA | 343b4202e553d58a76a36c60018588615bd4c30a |
| # of CPUs | 24 |
| # of Goroutines | 1503 |

Jira issue: CRDB-60165