cockroachdb / cockroachdb/cockroach

Sentry: panic.go:791: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:791 | runtime.p...

Open
#174,402 3 comments 0 reactions 0 assignees View on GitHub
branch-release-25.2 C-bug O-sentry T-kv X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7702035337/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7702035337/?referrer=webhooks_plugin)

Panic Message:

```
panic.go:791: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| runtime.panicmem
| GOROOT/src/runtime/panic.go:262
| runtime.sigpanic
| GOROOT/src/runtime/signal_unix.go:917
| github.com/cockroachdb/cockroach/pkg/kv/kvpb.actualSpan
| pkg/kv/kvpb/batch.go:574
| github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*BatchRequest).RefreshSpanIterate
| pkg/kv/kvpb/batch.go:556
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).appendRefreshSpans
| pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:654
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked
| pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:179
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked
| pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:144
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked
| pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:334
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnWriteBuffer).SendLocked
| pkg/kv/kvclient/kvcoord/txn_interceptor_write_buffer.go:208
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked
| pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:112
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked
| pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:265
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
| pkg/kv/kvclient/kvcoord/txn_coord_sender.go:558
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
| pkg/kv/db.go:1149
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
| pkg/kv/txn.go:1351
| github.com/cockroachdb/cockroach/pkg/kv.sendAndFill
| pkg/kv/db.go:941
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Run
| pkg/kv/txn.go:843
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).GetForUpdate
| pkg/kv/txn.go:582
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.conditionalGetDescValueFromDB
| pkg/kv/kvserver/replica_command.go:3497
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).AdminMerge.func1
| pkg/kv/kvserver/replica_command.go:757
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).AdminMerge
| pkg/kv/kvserver/replica_command.go:985
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*mergeQueue).process
| pkg/kv/kvserver/merge_queue.go:402
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1
| pkg/kv/kvserver/queue.go:974
| github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout
| pkg/util/timeutil/timeout.go:28
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica
| pkg/kv/kvserver/queue.go:967
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplicasInPurgatory.func1.2
| pkg/kv/kvserver/queue.go:1315
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask
| pkg/util/stop/stopper.go:317
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplicasInPurgatory.func1
| pkg/kv/kvserver/queue.go:1310
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplicasInPurgatory
| pkg/kv/kvserver/queue.go:1327
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).addToPurgatoryLocked.func2
| pkg/kv/kvserver/queue.go:1255
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:500
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) runtime.errorString
-- report composition:
runtime.errorString
panic.go:791: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[pkg/util/stop/stopper.go#L499-L501](pkg/util/stop/stopper.go#L499-L501)
[pkg/kv/kvserver/queue.go#L1254-L1256](pkg/kv/kvserver/queue.go#L1254-L1256)
[pkg/kv/kvserver/queue.go#L1326-L1328](pkg/kv/kvserver/queue.go#L1326-L1328)
[pkg/kv/kvserver/queue.go#L1309-L1311](pkg/kv/kvserver/queue.go#L1309-L1311)
[pkg/util/stop/stopper.go#L316-L318](pkg/util/stop/stopper.go#L316-L318)
[pkg/kv/kvserver/queue.go#L1314-L1316](pkg/kv/kvserver/queue.go#L1314-L1316)
[pkg/kv/kvserver/queue.go#L966-L968](pkg/kv/kvserver/queue.go#L966-L968)
[pkg/util/timeutil/timeout.go#L27-L29](pkg/util/timeutil/timeout.go#L27-L29)
[pkg/kv/kvserver/queue.go#L973-L975](pkg/kv/kvserver/queue.go#L973-L975)
[pkg/kv/kvserver/merge_queue.go#L401-L403](pkg/kv/kvserver/merge_queue.go#L401-L403)
[pkg/kv/kvserver/replica_command.go#L984-L986](pkg/kv/kvserver/replica_command.go#L984-L986)
[pkg/kv/kvserver/replica_command.go#L756-L758](pkg/kv/kvserver/replica_command.go#L756-L758)
[pkg/kv/kvserver/replica_command.go#L3496-L3498](pkg/kv/kvserver/replica_command.go#L3496-L3498)
[pkg/kv/txn.go#L581-L583](pkg/kv/txn.go#L581-L583)
[pkg/kv/txn.go#L842-L844](pkg/kv/txn.go#L842-L844)
[pkg/kv/db.go#L940-L942](pkg/kv/db.go#L940-L942)
[pkg/kv/txn.go#L1350-L1352](pkg/kv/txn.go#L1350-L1352)
[pkg/kv/db.go#L1148-L1150](pkg/kv/db.go#L1148-L1150)
[pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L557-L559](pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L557-L559)
[pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go#L264-L266](pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go#L264-L266)
[pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go#L111-L113](pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go#L111-L113)
[pkg/kv/kvclient/kvcoord/txn_interceptor_write_buffer.go#L207-L209](pkg/kv/kvclient/kvcoord/txn_interceptor_write_buffer.go#L207-L209)
[pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go#L333-L335](pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go#L333-L335)
[pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go#L143-L145](pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go#L143-L145)
[pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L178-L180](pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L178-L180)
[pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L653-L655](pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L653-L655)
[pkg/kv/kvpb/batch.go#L555-L557](pkg/kv/kvpb/batch.go#L555-L557)
[pkg/kv/kvpb/batch.go#L573-L575](pkg/kv/kvpb/batch.go#L573-L575)
[GOROOT/src/runtime/signal_unix.go#L916-L918](GOROOT/src/runtime/signal_unix.go#L916-L918)
[GOROOT/src/runtime/panic.go#L261-L263](GOROOT/src/runtime/panic.go#L261-L263)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 500
pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).addToPurgatoryLocked.func2 at line 1255
pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).processReplicasInPurgatory at line 1327
pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).processReplicasInPurgatory.func1 at line 1310
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTask at line 317
pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).processReplicasInPurgatory.func1.2 at line 1315
pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).processReplica at line 967
pkg/util/timeutil/timeout.go in pkg/util/timeutil.RunWithTimeout at line 28
pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).processReplica.func1 at line 974
pkg/kv/kvserver/merge_queue.go in pkg/kv/kvserver.(*mergeQueue).process at line 402
pkg/kv/kvserver/replica_command.go in pkg/kv/kvserver.(*Replica).AdminMerge at line 985
pkg/kv/kvserver/replica_command.go in pkg/kv/kvserver.(*Replica).AdminMerge.func1 at line 757
pkg/kv/kvserver/replica_command.go in pkg/kv/kvserver.conditionalGetDescValueFromDB at line 3497
pkg/kv/txn.go in pkg/kv.(*Txn).GetForUpdate at line 582
pkg/kv/txn.go in pkg/kv.(*Txn).Run at line 843
pkg/kv/db.go in pkg/kv.sendAndFill at line 941
pkg/kv/txn.go in pkg/kv.(*Txn).Send at line 1351
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1149
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send at line 558
pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go in pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked at line 265
pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go in pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked at line 112
pkg/kv/kvclient/kvcoord/txn_interceptor_write_buffer.go in pkg/kv/kvclient/kvcoord.(*txnWriteBuffer).SendLocked at line 208
pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go in pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked at line 334
pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go in pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked at line 144
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked at line 179
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).appendRefreshSpans at line 654
pkg/kv/kvpb/batch.go in pkg/kv/kvpb.(*BatchRequest).RefreshSpanIterate at line 556
pkg/kv/kvpb/batch.go in pkg/kv/kvpb.actualSpan at line 574
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 917
GOROOT/src/runtime/panic.go in runtime.panicmem at line 262
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v25.2.0 |
| Go Version | go1.23.7 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.2.0 |
| Cockroach SHA | 22cec2232cf3ab2a0e5f93cd35aa1217d62a4400 |
| # of CPUs | 8 |
| # of Goroutines | 618 |

Jira issue: CRDB-67676

Contributor guide

Open the contributing guide

Research direction

Start at pkg/kv/kvpb/batch.go:574 in actualSpan, then trace RefreshSpanIterate and txnSpanRefresher.appendRefreshSpans using the attached stack trace. Review the start-single-node v25.2.0 context and the AdminMerge path in pkg/kv/kvserver/replica_command.go. Done means the nil dereference is prevented and the affected path has regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.