Sentry: store_raft.go:762: log.Fatal: failed searching for sideloaded entries: matching pattern × on dir ×: sideloaded file not found (1) attached stack trace -- stack trace: | github.com/cockro...

Open

#166,807 opened on Mar 26, 2026 1 comment 0 reactions 0 assignees View on GitHub

branch-release-25.3 C-bug O-sentry T-kv X-blathers-triaged

Dominant language: Go
Stars: 32.5k
Forks: 4.1k
PR merge metrics: PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7365345193/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7365345193/?referrer=webhooks_plugin)

Panic Message:

```
store_raft.go:762: log.Fatal: failed searching for sideloaded entries: matching pattern × on dir ×: sideloaded file not found
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:762
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:411
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:302
| runtime.goexit
| src/runtime/asm_arm64.s:1223
Wraps: (2) secondary error attachment
| failed searching for sideloaded entries: matching pattern × on dir ×: sideloaded file not found
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).stageTruncation
| | pkg/kv/kvserver/replica_app_batch.go:535
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).runPostAddTriggersReplicaOnly
| | pkg/kv/kvserver/replica_app_batch.go:417
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| | pkg/kv/kvserver/replica_app_batch.go:157
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| | pkg/kv/kvserver/apply/cmd.go:187
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| | pkg/kv/kvserver/apply/task.go:279
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| | pkg/kv/kvserver/apply/task.go:246
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | pkg/kv/kvserver/replica_raft.go:1234
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | pkg/kv/kvserver/replica_raft.go:869
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | pkg/kv/kvserver/store_raft.go:761
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | pkg/kv/kvserver/scheduler.go:411
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | pkg/kv/kvserver/scheduler.go:302
| Wraps: (2) failed searching for sideloaded entries
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*DiskSideloadStorage).forEach
| | pkg/kv/kvserver/logstore/sideload_disk.go:273
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*DiskSideloadStorage).Stats
| | pkg/kv/kvserver/logstore/sideload_disk.go:225
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).stageTruncation
| | pkg/kv/kvserver/replica_app_batch.go:532
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).runPostAddTriggersReplicaOnly
| | pkg/kv/kvserver/replica_app_batch.go:417
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| | pkg/kv/kvserver/replica_app_batch.go:157
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| | pkg/kv/kvserver/apply/cmd.go:187
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| | pkg/kv/kvserver/apply/task.go:279
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| | pkg/kv/kvserver/apply/task.go:246
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | pkg/kv/kvserver/replica_raft.go:1234
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | pkg/kv/kvserver/replica_raft.go:869
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | pkg/kv/kvserver/store_raft.go:761
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | pkg/kv/kvserver/scheduler.go:411
| | [...repeated from below...]
| Wraps: (4) matching pattern × on dir ×
| Wraps: (5) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.init
| | pkg/kv/kvserver/logstore/sideload.go:21
| | runtime.doInit1
| | GOROOT/src/runtime/proc.go:7339
| | runtime.doInit
| | GOROOT/src/runtime/proc.go:7306
| | runtime.main
| | GOROOT/src/runtime/proc.go:254
| | runtime.goexit
| | src/runtime/asm_arm64.s:1223
| Wraps: (6) sideloaded file not found
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Wraps: (3) log.Fatal: failed searching for sideloaded entries: matching pattern × on dir ×: sideloaded file not found
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).stageTruncation
| | pkg/kv/kvserver/replica_app_batch.go:535
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).runPostAddTriggersReplicaOnly
| | pkg/kv/kvserver/replica_app_batch.go:417
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| | pkg/kv/kvserver/replica_app_batch.go:157
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| | pkg/kv/kvserver/apply/cmd.go:187
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| | pkg/kv/kvserver/apply/task.go:279
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| | pkg/kv/kvserver/apply/task.go:246
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | pkg/kv/kvserver/replica_raft.go:1234
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | pkg/kv/kvserver/replica_raft.go:869
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | pkg/kv/kvserver/store_raft.go:761
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | pkg/kv/kvserver/scheduler.go:411
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | pkg/kv/kvserver/scheduler.go:302
| Wraps: (2) failed searching for sideloaded entries
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*DiskSideloadStorage).forEach
| | pkg/kv/kvserver/logstore/sideload_disk.go:273
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*DiskSideloadStorage).Stats
| | pkg/kv/kvserver/logstore/sideload_disk.go:225
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).stageTruncation
| | pkg/kv/kvserver/replica_app_batch.go:532
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).runPostAddTriggersReplicaOnly
| | pkg/kv/kvserver/replica_app_batch.go:417
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| | pkg/kv/kvserver/replica_app_batch.go:157
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| | pkg/kv/kvserver/apply/cmd.go:187
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| | pkg/kv/kvserver/apply/task.go:279
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| | pkg/kv/kvserver/apply/task.go:246
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | pkg/kv/kvserver/replica_raft.go:1234
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | pkg/kv/kvserver/replica_raft.go:869
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | pkg/kv/kvserver/store_raft.go:761
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | pkg/kv/kvserver/scheduler.go:411
| | [...repeated from below...]
| Wraps: (4) matching pattern × on dir ×
| Wraps: (5) attached stack trace
| -- stack tra...
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1222-L1224](src/runtime/asm_arm64.s#L1222-L1224)
[pkg/kv/kvserver/scheduler.go#L301-L303](pkg/kv/kvserver/scheduler.go#L301-L303)
[pkg/kv/kvserver/scheduler.go#L410-L412](pkg/kv/kvserver/scheduler.go#L410-L412)
[pkg/kv/kvserver/store_raft.go#L761-L763](pkg/kv/kvserver/store_raft.go#L761-L763)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 411
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 762
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v25.3.1 |
| Go Version | go1.23.11 X:nocoverageredesign |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v25.3.1 |
| Cockroach SHA | 092bace42e2cb3784a76705117251577939bd869 |
| # of CPUs | 10 |
| # of Goroutines | 533 |

Jira issue: CRDB-62095

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Sentry: store_raft.go:762: log.Fatal: failed searching for sideloaded entries: matching pattern × on dir ×: sideloaded file not found (1) attached stack trace -- stack trace: | github.com/cockro...

Description

Contributor guide

Assessment

Get new issues in your inbox