cockroachdb / cockroachdb/cockroach

Sentry: panic.go:770: panic: must not be holding latches (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:770 | github.com/cockroachdb/cockroach/pkg/ut...

Open
#172,173 1 comment 0 reactions 0 assignees View on GitHub
branch-release-24.2 C-bug O-sentry T-kv X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7587502303/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7587502303/?referrer=webhooks_plugin)

Panic Message:

```
panic.go:770: panic: must not be holding latches
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:770
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226
| runtime.gopanic
| GOROOT/src/runtime/panic.go:770
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:104
| runtime.gopanic
| GOROOT/src/runtime/panic.go:770
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:302
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:239
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:453
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:177
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:188
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:197
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| github.com/cockroachdb/cockroach/pkg/server/node.go:1518
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
| github.com/cockroachdb/cockroach/pkg/server/node.go:1719
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:703
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:92
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:165
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:132
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:332
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:130
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:713
| github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:223
| github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:898
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:783
| github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:906
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:209
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:187
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2752
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2200
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1749
Wraps: (2) panic: must not be holding latches
| this is tracked in github.com/cockroachdb/cockroach/issues/77663; please comment if seen
| eval_kind=0, holding_latches=true, branch=1, first_iteration=true, stack=
| goroutine 3675391 [running]:
| runtime/debug.Stack()
| GOROOT/src/runtime/debug/stack.go:24 +0x5e
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc027fd7d60, {0x8370af0, 0xc043912150}, 0xc0a00d4750, 0x1)
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:305 +0xae8
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc027fd7d60, {0x8370af0, 0xc043912150}, 0x5f835c0?, {0xc02f9bc120, {0x18be3779b2918da4, 0x0}, 0x0, 0x0, 0x0, ...}, ...)
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:239 +0x2f2
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc0401b1108, {0x8370af0, 0xc043912150}, 0xc02f9aa8c0, 0x6e79c88)
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:453 +0x34c
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc0401b1108, {0x8370af0?, 0xc043912120?}, 0xc02f9aa8c0)
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:177 +0x6d0
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc002bd8008, {0x8370af0?, 0xc0439120f0?}, 0xc02f9aa8c0)
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:188 +0x83c
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc0018f5180, {0x8370af0, 0xc0439120f0}, 0xc02f9aa8c0)
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:197 +0x7e
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc007142c08, {0x8370af0?, 0xc0439120c0?}, {0xc035daff10?}, 0xc02f9aa8c0)
| github.com/cockroachdb/cockroach/pkg/server/node.go:1518 +0x5b4
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc007142c08, {0x8370af0, 0xc043912060}, 0xc02f9aa8c0)
| github.com/cockroachdb/cockroach/pkg/server/node.go:1719 +0x2a6
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0x8370af0?, 0xc043912060?}, {0x695a960?, 0xc02f9aa8c0?})
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:703 +0x45
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x8370af0, 0xc043912060}, {0x695a960, 0xc02f9aa8c0}, 0xc00eeb0480, 0xc00edff620)
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:92 +0x4b2
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x8370af0?, 0xc043912060?}, {0x695a960?, 0xc02f9aa8c0?})
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814 +0x34
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x8370af0, 0xc043912060}, {0x695a960, 0xc02f9aa8c0}, 0xc035db0648?, 0xc00eeb04a0)
| github.com/cock...
```

Stacktrace (expand for inline code snippets):

[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvclient/kvcoord/dist_sender.go#L1748-L1750](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvclient/kvcoord/dist_sender.go#L1748-L1750)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvclient/kvcoord/dist_sender.go#L2199-L2201](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvclient/kvcoord/dist_sender.go#L2199-L2201)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvclient/kvcoord/dist_sender.go#L2751-L2753](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvclient/kvcoord/dist_sender.go#L2751-L2753)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvclient/kvcoord/transport.go#L186-L188](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvclient/kvcoord/transport.go#L186-L188)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvclient/kvcoord/transport.go#L208-L210](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvclient/kvcoord/transport.go#L208-L210)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L905-L907](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L905-L907)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L782-L784](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L782-L784)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L897-L899](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L897-L899)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L222-L224](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L222-L224)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L712-L714](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L712-L714)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L813-L815](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L813-L815)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L129-L131](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L129-L131)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/util/stop/stopper.go#L331-L333](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/util/stop/stopper.go#L331-L333)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L131-L133](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L131-L133)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L813-L815](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L813-L815)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L164-L166](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L164-L166)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L813-L815](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L813-L815)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L91-L93](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L91-L93)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L702-L704](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/rpc/pkg/rpc/context.go#L702-L704)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/server/node.go#L1718-L1720](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/server/node.go#L1718-L1720)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/server/node.go#L1517-L1519](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/server/node.go#L1517-L1519)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L196-L198](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L196-L198)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L187-L189](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L187-L189)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L176-L178](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L176-L178)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L452-L454](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L452-L454)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/concurrency/concurrency_manager.go#L238-L240](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/concurrency/concurrency_manager.go#L238-L240)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/concurrency/concurrency_manager.go#L301-L303](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/concurrency/concurrency_manager.go#L301-L303)
[GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L103-L105](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L103-L105)
[GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771)
[https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/util/stop/stopper.go#L225-L227](https://github.com/cockroachdb/cockroach/blob/046116ac37efed5175dd59aaaadf91d1f7c12010/pkg/util/stop/stopper.go#L225-L227)
[GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771)

```
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges at line 1749
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 2200
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2752
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 187
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 209
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 906
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 783
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 898
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewContext.ClientInterceptor.func8 at line 223
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 713
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 814
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 130
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 332
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 132
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 814
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 165
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 814
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 92
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 703
pkg/server/node.go in pkg/server.(*Node).Batch at line 1719
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1518
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 197
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 188
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 177
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 453
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq at line 239
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard at line 302
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1 at line 104
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).recover at line 226
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v24.2.10 |
| Go Version | go1.22.5 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.2.10 |
| Cockroach SHA | 046116ac37efed5175dd59aaaadf91d1f7c12010 |
| # of CPUs | 4 |
| # of Goroutines | 530 |

Jira issue: CRDB-65319

Contributor guide

Open the contributing guide

Research direction

Start with pkg/kv/kvserver/concurrency/concurrency_manager.go at SequenceReq and sequenceReqWithGuard (lines 239 and 302-305), then trace the calls through replica_send.go and the linked issue #77663. Use the Sentry stack and referenced dist_sender.go locations to determine why latches are held during the panic; done means the reported path no longer triggers this panic and has regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.