cockroachdb / cockroachdb/cockroach

Sentry: panic.go:791: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:791 | github.co...

Open
#169,466 1 comment 0 reactions 0 assignees View on GitHub
branch-release-25.3 C-bug O-sentry T-kv X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7452636036/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7452636036/?referrer=webhooks_plugin)

Panic Message:

```
panic.go:791: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1
| pkg/kv/kvserver/store_send.go:104
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| runtime.panicmem
| GOROOT/src/runtime/panic.go:262
| runtime.sigpanic
| GOROOT/src/runtime/signal_unix.go:917
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*latchList).remove
| pkg/kv/kvserver/spanlatch/list.go:40
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*scopedManager).flushReadSetLocked
| pkg/kv/kvserver/spanlatch/manager.go:436
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).snapshotLocked
| pkg/kv/kvserver/spanlatch/manager.go:422
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).sequence
| pkg/kv/kvserver/spanlatch/manager.go:392
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).Acquire
| pkg/kv/kvserver/spanlatch/manager.go:251
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*latchManagerImpl).Acquire
| pkg/kv/kvserver/concurrency/latch_manager.go:23
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard
| pkg/kv/kvserver/concurrency/concurrency_manager.go:354
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq
| pkg/kv/kvserver/concurrency/concurrency_manager.go:302
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| pkg/kv/kvserver/replica_send.go:465
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| pkg/kv/kvserver/replica_send.go:189
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| pkg/kv/kvserver/store_send.go:188
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
| pkg/kv/kvserver/stores.go:199
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| pkg/server/node.go:1665
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
| pkg/server/node.go:1858
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
| pkg/rpc/context.go:735
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:96
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
| pkg/rpc/context.go:178
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.gatewayRequestRecoveryInterceptor
| pkg/rpc/metrics.go:462
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4
| pkg/rpc/metrics.go:415
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
| pkg/rpc/context.go:136
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:349
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
| pkg/rpc/context.go:134
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
| pkg/rpc/context.go:846
Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) runtime.errorString
-- report composition:
runtime.errorString
panic.go:791: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/context.go#L133-L135](pkg/rpc/context.go#L133-L135)
[pkg/util/stop/stopper.go#L348-L350](pkg/util/stop/stopper.go#L348-L350)
[pkg/rpc/context.go#L135-L137](pkg/rpc/context.go#L135-L137)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/metrics.go#L414-L416](pkg/rpc/metrics.go#L414-L416)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/metrics.go#L461-L463](pkg/rpc/metrics.go#L461-L463)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/rpc/context.go#L177-L179](pkg/rpc/context.go#L177-L179)
[pkg/rpc/context.go#L845-L847](pkg/rpc/context.go#L845-L847)
[pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L95-L97](pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L95-L97)
[pkg/rpc/context.go#L734-L736](pkg/rpc/context.go#L734-L736)
[pkg/server/node.go#L1857-L1859](pkg/server/node.go#L1857-L1859)
[pkg/server/node.go#L1664-L1666](pkg/server/node.go#L1664-L1666)
[pkg/kv/kvserver/stores.go#L198-L200](pkg/kv/kvserver/stores.go#L198-L200)
[pkg/kv/kvserver/store_send.go#L187-L189](pkg/kv/kvserver/store_send.go#L187-L189)
[pkg/kv/kvserver/replica_send.go#L188-L190](pkg/kv/kvserver/replica_send.go#L188-L190)
[pkg/kv/kvserver/replica_send.go#L464-L466](pkg/kv/kvserver/replica_send.go#L464-L466)
[pkg/kv/kvserver/concurrency/concurrency_manager.go#L301-L303](pkg/kv/kvserver/concurrency/concurrency_manager.go#L301-L303)
[pkg/kv/kvserver/concurrency/concurrency_manager.go#L353-L355](pkg/kv/kvserver/concurrency/concurrency_manager.go#L353-L355)
[pkg/kv/kvserver/concurrency/latch_manager.go#L22-L24](pkg/kv/kvserver/concurrency/latch_manager.go#L22-L24)
[pkg/kv/kvserver/spanlatch/manager.go#L250-L252](pkg/kv/kvserver/spanlatch/manager.go#L250-L252)
[pkg/kv/kvserver/spanlatch/manager.go#L391-L393](pkg/kv/kvserver/spanlatch/manager.go#L391-L393)
[pkg/kv/kvserver/spanlatch/manager.go#L421-L423](pkg/kv/kvserver/spanlatch/manager.go#L421-L423)
[pkg/kv/kvserver/spanlatch/manager.go#L435-L437](pkg/kv/kvserver/spanlatch/manager.go#L435-L437)
[pkg/kv/kvserver/spanlatch/list.go#L39-L41](pkg/kv/kvserver/spanlatch/list.go#L39-L41)
[GOROOT/src/runtime/signal_unix.go#L916-L918](GOROOT/src/runtime/signal_unix.go#L916-L918)
[GOROOT/src/runtime/panic.go#L261-L263](GOROOT/src/runtime/panic.go#L261-L263)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)
[pkg/kv/kvserver/store_send.go#L103-L105](pkg/kv/kvserver/store_send.go#L103-L105)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)

```
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 134
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 349
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 136
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/metrics.go in pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4 at line 415
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/metrics.go in pkg/rpc.gatewayRequestRecoveryInterceptor at line 462
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 178
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 846
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 96
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 735
pkg/server/node.go in pkg/server.(*Node).Batch at line 1858
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1665
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 199
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 188
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 189
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 465
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq at line 302
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard at line 354
pkg/kv/kvserver/concurrency/latch_manager.go in pkg/kv/kvserver/concurrency.(*latchManagerImpl).Acquire at line 23
pkg/kv/kvserver/spanlatch/manager.go in pkg/kv/kvserver/spanlatch.(*Manager).Acquire at line 251
pkg/kv/kvserver/spanlatch/manager.go in pkg/kv/kvserver/spanlatch.(*Manager).sequence at line 392
pkg/kv/kvserver/spanlatch/manager.go in pkg/kv/kvserver/spanlatch.(*Manager).snapshotLocked at line 422
pkg/kv/kvserver/spanlatch/manager.go in pkg/kv/kvserver/spanlatch.(*scopedManager).flushReadSetLocked at line 436
pkg/kv/kvserver/spanlatch/list.go in pkg/kv/kvserver/spanlatch.(*latchList).remove at line 40
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 917
GOROOT/src/runtime/panic.go in runtime.panicmem at line 262
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1 at line 104
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v25.3.6 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.3.6 |
| Cockroach SHA | 7bad8e82ef2989b3cd657baca02567a892c1b5c3 |
| # of CPUs | 384 |
| # of Goroutines | 3168 |

Jira issue: CRDB-63463

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.