Sentry: replica.go:1957: log.Fatal: pebble: backing file 008913 error: open ×: operation not permitted (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver....

Open

#156,719 opened on Nov 1, 2025 1 comment 0 reactions 0 assignees View on GitHub

branch-release-25.3 C-bug O-sentry T-kv X-blathers-triaged

Dominant language: Go
Stars: 32.5k
Forks: 4.1k
PR merge metrics: PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/6989869468/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6989869468/?referrer=webhooks_plugin)

Panic Message:

```
replica.go:1957: log.Fatal: pebble: backing file 008913 error: open ×: operation not permitted
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked
| pkg/kv/kvserver/replica.go:1957
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1283
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:869
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:761
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:411
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:302
| runtime.goexit
| src/runtime/asm_arm64.s:1223
Wraps: (2) secondary error attachment
| pebble: backing file 008913 error: open ×: operation not permitted
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:468
| | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| | external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:191
| | github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
| | external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:84
| | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateTable
| | external/com_github_cockroachdb_pebble/file_cache.go:260
| | github.com/cockroachdb/pebble.(*fileCacheHandle).newIters
| | external/com_github_cockroachdb_pebble/file_cache.go:553
| | github.com/cockroachdb/pebble.(*levelIter).loadFile
| | external/com_github_cockroachdb_pebble/level_iter.go:568
| | github.com/cockroachdb/pebble.(*levelIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/level_iter.go:661
| | github.com/cockroachdb/pebble.(*mergingIter).seekGE
| | external/com_github_cockroachdb_pebble/merging_iter.go:969
| | github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGEStrict
| | external/com_github_cockroachdb_pebble/merging_iter.go:1044
| | github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/merging_iter.go:1035
| | github.com/cockroachdb/pebble.(*lazyCombinedIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/range_keys.go:618
| | github.com/cockroachdb/pebble.(*Iterator).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/iterator.go:1502
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
| | pkg/storage/pebble_iterator.go:343
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).get
| | pkg/storage/pebble_mvcc_scanner.go:584
| | github.com/cockroachdb/cockroach/pkg/storage.mvccGet
| | pkg/storage/mvcc.go:1583
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGetWithValueHeader
| | pkg/storage/mvcc.go:1508
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGet
| | pkg/storage/mvcc.go:1408
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGetProto
| | pkg/storage/mvcc.go:1104
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.StateLoader.LoadRaftTruncatedState
| | pkg/kv/kvserver/logstore/stateloader.go:111
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked
| | pkg/kv/kvserver/replica.go:1956
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | pkg/kv/kvserver/replica_raft.go:1283
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | pkg/kv/kvserver/replica_raft.go:869
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | pkg/kv/kvserver/store_raft.go:761
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | pkg/kv/kvserver/scheduler.go:411
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | pkg/kv/kvserver/scheduler.go:302
| Wraps: (2) pebble: backing file 008913 error
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/vfs.defaultFS.Open
| | external/com_github_cockroachdb_pebble/vfs/vfs.go:275
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Open
| | external/com_github_cockroachdb_pebble/vfs/disk_health.go:805
| | github.com/cockroachdb/pebble/vfs.(*enospcFS).Open
| | external/com_github_cockroachdb_pebble/vfs/disk_full.go:221
| | github.com/cockroachdb/cockroach/pkg/storage/fs.(*Env).Open
| | pkg/storage/fs/fs.go:363
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).vfsOpenForReading
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs.go:27
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).OpenForReading
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:304
| | github.com/cockroachdb/pebble.(*fileCacheHandle).openFile
| | external/com_github_cockroachdb_pebble/file_cache.go:209
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:466
| | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| | external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:191
| | github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
| | external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:84
| | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateTable
| | external/com_github_cockroachdb_pebble/file_cache.go:260
| | github.com/cockroachdb/pebble.(*fileCacheHandle).newIters
| | external/com_github_cockroachdb_pebble/file_cache.go:553
| | github.com/cockroachdb/pebble.(*levelIter).loadFile
| | external/com_github_cockroachdb_pebble/level_iter.go:568
| | github.com/cockroachdb/pebble.(*levelIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/level_iter.go:661
| | github.com/cockroachdb/pebble.(*mergingIter).seekGE
| | external/com_github_cockroachdb_pebble/merging_iter.go:969
| | github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGEStrict
| | external/com_github_cockroachdb_pebble/merging_iter.go:1044
| | github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/merging_iter.go:1035
| | github.com/cockroachdb/pebble.(*lazyCombinedIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/range_keys.go:618
| | github.com/cockroachdb/pebble.(*Iterator).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/iterator.go:1502
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
| | pkg/storage/pebble_iterator.go:343
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).get
| | pkg/storage/pebble_mvcc_scanner.go:584
| | github.com/cockroachdb/cockroach/pkg/storage.mvccGet
| | pkg/storage/mvcc.go:1583
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGetWithValueHeader
| | pkg/storage/mvcc.go:1508
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGet
| | pkg/storage/mvcc.go:1408
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGetProto
| | pkg/storage/mvcc.go:1104
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.StateLoader.LoadRaftTruncatedState
| | pkg/kv/kvserver/logstore/stateloader.go:111
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked
| | pkg/kv/kvserver/replica.go:1956
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| ...
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1222-L1224](src/runtime/asm_arm64.s#L1222-L1224)
[pkg/kv/kvserver/scheduler.go#L301-L303](pkg/kv/kvserver/scheduler.go#L301-L303)
[pkg/kv/kvserver/scheduler.go#L410-L412](pkg/kv/kvserver/scheduler.go#L410-L412)
[pkg/kv/kvserver/store_raft.go#L760-L762](pkg/kv/kvserver/store_raft.go#L760-L762)
[pkg/kv/kvserver/replica_raft.go#L868-L870](pkg/kv/kvserver/replica_raft.go#L868-L870)
[pkg/kv/kvserver/replica_raft.go#L1282-L1284](pkg/kv/kvserver/replica_raft.go#L1282-L1284)
[pkg/kv/kvserver/replica.go#L1956-L1958](pkg/kv/kvserver/replica.go#L1956-L1958)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 411
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 761
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 869
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1283
pkg/kv/kvserver/replica.go in pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked at line 1957
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.3.2 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | darwin arm64 |
| Distribution | CCL |
| Cockroach Release | v25.3.2 |
| Cockroach SHA | a5e060ee11ad11e1649e7569486b579e9da8c8d7 |
| # of CPUs | 10 |
| # of Goroutines | 638 |

Jira issue: CRDB-56141

Contributor guide

Open the contributing guide

Research direction

Start with pkg/kv/kvserver/replica.go at assertStateRaftMuLockedReplicaMuRLocked, then trace the cited calls through replica_raft.go, store_raft.go, and pkg/storage/logstore/stateloader.go. Use the Sentry report and Pebble file-cache stack to determine why opening backing file 008913 returns operation not permitted; done means identifying the cause and adding a validated fix or regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack: go
Domain: databases, distributed-systems
Issue type: Bug
Difficulty: 4/5
Estimated time: 3-5 days
Activity status: Stale
Clarity: Needs clarification
Newbie friendliness: 20/100

Sentry: replica.go:1957: log.Fatal: pebble: backing file 008913 error: open ×: operation not permitted (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver....

Description

Contributor guide

Research direction

Assessment

Get new issues in your inbox