cockroachdb / cockroachdb/cockroach

Sentry: sideload.go:21: loading sideloaded data: sideloaded file not found (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:791 | github.com/cockroachd...

Open
#151,611 1 comment 0 reactions 0 assignees View on GitHub
branch-release-25.2 C-bug O-sentry T-kv X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/6799580028/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6799580028/?referrer=webhooks_plugin)

Panic Message:

```
sideload.go:21: loading sideloaded data: sideloaded file not found
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| github.com/cockroachdb/cockroach/pkg/raft.LogSnapshot.slice
| pkg/raft/log.go:601
| github.com/cockroachdb/cockroach/pkg/raft.(*raftLog).slice
| pkg/raft/log.go:538
| github.com/cockroachdb/cockroach/pkg/raft.(*raftLog).entries
| pkg/raft/log.go:458
| github.com/cockroachdb/cockroach/pkg/raft.(*raft).maybeSendAppend
| pkg/raft/raft.go:717
| github.com/cockroachdb/cockroach/pkg/raft.(*raft).tickHeartbeat.(*raft).maybeUnpauseAndBcastAppend.func2
| pkg/raft/raft.go:973
| github.com/cockroachdb/cockroach/pkg/raft/tracker.(*ProgressTracker).Visit
| pkg/raft/tracker/progresstracker.go:106
| github.com/cockroachdb/cockroach/pkg/raft.(*raft).maybeUnpauseAndBcastAppend
| pkg/raft/raft.go:955
| github.com/cockroachdb/cockroach/pkg/raft.(*raft).tickHeartbeat
| pkg/raft/raft.go:1262
| github.com/cockroachdb/cockroach/pkg/raft.(*RawNode).Tick
| pkg/raft/rawnode.go:64
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).tick
| pkg/kv/kvserver/replica_raft.go:1509
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processTick
| pkg/kv/kvserver/store_raft.go:784
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:410
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:319
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:500
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.MaybeInlineSideloadedRaftCommand
| pkg/kv/kvserver/logstore/sideload.go:197
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.LoadEntries.func1
| pkg/kv/kvserver/logstore/logstore.go:681
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftlog.Visit
| pkg/kv/kvserver/raftlog/iterator.go:159
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.LoadEntries
| pkg/kv/kvserver/logstore/logstore.go:699
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaLogStorage).entriesLocked
| pkg/kv/kvserver/replica_raftlog.go:80
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaLogStorage).Entries
| pkg/kv/kvserver/replica_raftlog.go:46
| github.com/cockroachdb/cockroach/pkg/raft.LogSnapshot.slice
| pkg/raft/log.go:595
| github.com/cockroachdb/cockroach/pkg/raft.(*raftLog).slice
| pkg/raft/log.go:538
| github.com/cockroachdb/cockroach/pkg/raft.(*raftLog).entries
| pkg/raft/log.go:458
| github.com/cockroachdb/cockroach/pkg/raft.(*raft).maybeSendAppend
| pkg/raft/raft.go:717
| github.com/cockroachdb/cockroach/pkg/raft.(*raft).tickHeartbeat.(*raft).maybeUnpauseAndBcastAppend.func2
| pkg/raft/raft.go:973
| github.com/cockroachdb/cockroach/pkg/raft/tracker.(*ProgressTracker).Visit
| pkg/raft/tracker/progresstracker.go:106
| github.com/cockroachdb/cockroach/pkg/raft.(*raft).maybeUnpauseAndBcastAppend
| pkg/raft/raft.go:955
| github.com/cockroachdb/cockroach/pkg/raft.(*raft).tickHeartbeat
| pkg/raft/raft.go:1262
| github.com/cockroachdb/cockroach/pkg/raft.(*RawNode).Tick
| pkg/raft/rawnode.go:64
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).tick
| pkg/kv/kvserver/replica_raft.go:1509
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processTick
| pkg/kv/kvserver/store_raft.go:784
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:410
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:319
| [...repeated from below...]
Wraps: (3) loading sideloaded data
Wraps: (4) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.init
| pkg/kv/kvserver/logstore/sideload.go:21
| runtime.doInit1
| GOROOT/src/runtime/proc.go:7321
| runtime.doInit
| GOROOT/src/runtime/proc.go:7288
| runtime.main
| GOROOT/src/runtime/proc.go:254
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (5) sideloaded file not found
Error types: (1) *withstack.withStack (2) *withstack.withStack (3) *errutil.withPrefix (4) *withstack.withStack (5) *errutil.leafError
-- report composition:
*errutil.leafError: sideloaded file not found
sideload.go:21: *withstack.withStack (top exception)
*errutil.withPrefix: loading sideloaded data
sideload.go:197: *withstack.withStack (1)
panic.go:791: *withstack.withStack (2)
(check the extra data payloads)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[pkg/util/stop/stopper.go#L499-L501](pkg/util/stop/stopper.go#L499-L501)
[pkg/kv/kvserver/scheduler.go#L318-L320](pkg/kv/kvserver/scheduler.go#L318-L320)
[pkg/kv/kvserver/scheduler.go#L409-L411](pkg/kv/kvserver/scheduler.go#L409-L411)
[pkg/kv/kvserver/store_raft.go#L783-L785](pkg/kv/kvserver/store_raft.go#L783-L785)
[pkg/kv/kvserver/replica_raft.go#L1508-L1510](pkg/kv/kvserver/replica_raft.go#L1508-L1510)
[pkg/raft/rawnode.go#L63-L65](pkg/raft/rawnode.go#L63-L65)
[pkg/raft/raft.go#L1261-L1263](pkg/raft/raft.go#L1261-L1263)
[pkg/raft/raft.go#L954-L956](pkg/raft/raft.go#L954-L956)
[pkg/raft/tracker/progresstracker.go#L105-L107](pkg/raft/tracker/progresstracker.go#L105-L107)
[pkg/raft/raft.go#L972-L974](pkg/raft/raft.go#L972-L974)
[pkg/raft/raft.go#L716-L718](pkg/raft/raft.go#L716-L718)
[pkg/raft/log.go#L457-L459](pkg/raft/log.go#L457-L459)
[pkg/raft/log.go#L537-L539](pkg/raft/log.go#L537-L539)
[pkg/raft/log.go#L600-L602](pkg/raft/log.go#L600-L602)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)
[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[pkg/util/stop/stopper.go#L499-L501](pkg/util/stop/stopper.go#L499-L501)
[pkg/kv/kvserver/scheduler.go#L318-L320](pkg/kv/kvserver/scheduler.go#L318-L320)
[pkg/kv/kvserver/scheduler.go#L409-L411](pkg/kv/kvserver/scheduler.go#L409-L411)
[pkg/kv/kvserver/store_raft.go#L783-L785](pkg/kv/kvserver/store_raft.go#L783-L785)
[pkg/kv/kvserver/replica_raft.go#L1508-L1510](pkg/kv/kvserver/replica_raft.go#L1508-L1510)
[pkg/raft/rawnode.go#L63-L65](pkg/raft/rawnode.go#L63-L65)
[pkg/raft/raft.go#L1261-L1263](pkg/raft/raft.go#L1261-L1263)
[pkg/raft/raft.go#L954-L956](pkg/raft/raft.go#L954-L956)
[pkg/raft/tracker/progresstracker.go#L105-L107](pkg/raft/tracker/progresstracker.go#L105-L107)
[pkg/raft/raft.go#L972-L974](pkg/raft/raft.go#L972-L974)
[pkg/raft/raft.go#L716-L718](pkg/raft/raft.go#L716-L718)
[pkg/raft/log.go#L457-L459](pkg/raft/log.go#L457-L459)
[pkg/raft/log.go#L537-L539](pkg/raft/log.go#L537-L539)
[pkg/raft/log.go#L594-L596](pkg/raft/log.go#L594-L596)
[pkg/kv/kvserver/replica_raftlog.go#L45-L47](pkg/kv/kvserver/replica_raftlog.go#L45-L47)
[pkg/kv/kvserver/replica_raftlog.go#L79-L81](pkg/kv/kvserver/replica_raftlog.go#L79-L81)
[pkg/kv/kvserver/logstore/logstore.go#L698-L700](pkg/kv/kvserver/logstore/logstore.go#L698-L700)
[pkg/kv/kvserver/raftlog/iterator.go#L158-L160](pkg/kv/kvserver/raftlog/iterator.go#L158-L160)
[pkg/kv/kvserver/logstore/logstore.go#L680-L682](pkg/kv/kvserver/logstore/logstore.go#L680-L682)
[pkg/kv/kvserver/logstore/sideload.go#L196-L198](pkg/kv/kvserver/logstore/sideload.go#L196-L198)
[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[GOROOT/src/runtime/proc.go#L253-L255](GOROOT/src/runtime/proc.go#L253-L255)
[GOROOT/src/runtime/proc.go#L7287-L7289](GOROOT/src/runtime/proc.go#L7287-L7289)
[GOROOT/src/runtime/proc.go#L7320-L7322](GOROOT/src/runtime/proc.go#L7320-L7322)
[pkg/kv/kvserver/logstore/sideload.go#L20-L22](pkg/kv/kvserver/logstore/sideload.go#L20-L22)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 500
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 319
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 410
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processTick at line 784
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).tick at line 1509
pkg/raft/rawnode.go in pkg/raft.(*RawNode).Tick at line 64
pkg/raft/raft.go in pkg/raft.(*raft).tickHeartbeat at line 1262
pkg/raft/raft.go in pkg/raft.(*raft).maybeUnpauseAndBcastAppend at line 955
pkg/raft/tracker/progresstracker.go in pkg/raft/tracker.(*ProgressTracker).Visit at line 106
pkg/raft/raft.go in pkg/raft.(*raft).tickHeartbeat.(*raft).maybeUnpauseAndBcastAppend.func2 at line 973
pkg/raft/raft.go in pkg/raft.(*raft).maybeSendAppend at line 717
pkg/raft/log.go in pkg/raft.(*raftLog).entries at line 458
pkg/raft/log.go in pkg/raft.(*raftLog).slice at line 538
pkg/raft/log.go in pkg/raft.LogSnapshot.slice at line 601
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
src/runtime/asm_amd64.s in runtime.goexit at line 1700
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 500
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 319
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 410
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processTick at line 784
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).tick at line 1509
pkg/raft/rawnode.go in pkg/raft.(*RawNode).Tick at line 64
pkg/raft/raft.go in pkg/raft.(*raft).tickHeartbeat at line 1262
pkg/raft/raft.go in pkg/raft.(*raft).maybeUnpauseAndBcastAppend at line 955
pkg/raft/tracker/progresstracker.go in pkg/raft/tracker.(*ProgressTracker).Visit at line 106
pkg/raft/raft.go in pkg/raft.(*raft).tickHeartbeat.(*raft).maybeUnpauseAndBcastAppend.func2 at line 973
pkg/raft/raft.go in pkg/raft.(*raft).maybeSendAppend at line 717
pkg/raft/log.go in pkg/raft.(*raftLog).entries at line 458
pkg/raft/log.go in pkg/raft.(*raftLog).slice at line 538
pkg/raft/log.go in pkg/raft.LogSnapshot.slice at line 595
pkg/kv/kvserver/replica_raftlog.go in pkg/kv/kvserver.(*replicaLogStorage).Entries at line 46
pkg/kv/kvserver/replica_raftlog.go in pkg/kv/kvserver.(*replicaLogStorage).entriesLocked at line 80
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.LoadEntries at line 699
pkg/kv/kvserver/raftlog/iterator.go in pkg/kv/kvserver/raftlog.Visit at line 159
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.LoadEntries.func1 at line 681
pkg/kv/kvserver/logstore/sideload.go in pkg/kv/kvserver/logstore.MaybeInlineSideloadedRaftCommand at line 197
src/runtime/asm_amd64.s in runtime.goexit at line 1700
GOROOT/src/runtime/proc.go in runtime.main at line 254
GOROOT/src/runtime/proc.go in runtime.doInit at line 7288
GOROOT/src/runtime/proc.go in runtime.doInit1 at line 7321
pkg/kv/kvserver/logstore/sideload.go in pkg/kv/kvserver/logstore.init at line 21
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.2.1 |
| Go Version | go1.23.7 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.2.1 |
| Cockroach SHA | f9f183de9d58b822d6cf4c12c560e1fbbc70707f |
| # of CPUs | 16 |
| # of Goroutines | 737 |

Jira issue: CRDB-53358

Contributor guide

Open the contributing guide

Research direction

Start with pkg/kv/kvserver/logstore/sideload.go, especially MaybeInlineSideloadedRaftCommand and its initialization, then trace how logstore.LoadEntries reaches pkg/raft/log.go. Use the Sentry stack trace and the v25.2.1 context to understand the missing sideloaded file case. Done means the reported raft path no longer panics when the sideloaded file is absent, with verification covering the affected path.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.