Sentry: replica.go:2020: log.Dev.Fatal: pebble: backing file 000093 error: open ×: operation not permitted (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvser...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7557176235/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7557176235/?referrer=webhooks_plugin)

Panic Message:

```
replica.go:2020: log.Dev.Fatal: pebble: backing file 000093 error: open ×: operation not permitted
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked
| pkg/kv/kvserver/replica.go:2020
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1296
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:884
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:760
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:421
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:314
| runtime.goexit
| src/runtime/asm_arm64.s:1268
Wraps: (2) secondary error attachment
| pebble: backing file 000093 error: open ×: operation not permitted
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:499
| | [...repeated from below...]
| Wraps: (2) pebble: backing file 000093 error
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/vfs.defaultFS.Open
| | external/com_github_cockroachdb_pebble/vfs/vfs.go:275
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Open
| | external/com_github_cockroachdb_pebble/vfs/disk_health.go:815
| | github.com/cockroachdb/pebble/vfs.(*enospcFS).Open
| | external/com_github_cockroachdb_pebble/vfs/disk_full.go:221
| | github.com/cockroachdb/cockroach/pkg/storage/fs.(*Env).Open
| | pkg/storage/fs/fs.go:384
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).localOpenForReading
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs.go:127
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).OpenForReading
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:283
| | github.com/cockroachdb/pebble.(*fileCacheHandle).openFile
| | external/com_github_cockroachdb_pebble/file_cache.go:210
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:497
| | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| | external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:234
| | github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
| | external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:92
| | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateBlob
| | external/com_github_cockroachdb_pebble/file_cache.go:283
| | github.com/cockroachdb/pebble.(*fileCacheHandle).GetValueReader
| | external/com_github_cockroachdb_pebble/file_cache.go:409
| | github.com/cockroachdb/pebble/sstable/blob.(*ValueFetcher).retrieve
| | external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go:190
| | github.com/cockroachdb/pebble/sstable/blob.(*ValueFetcher).FetchHandle
| | external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go:126
| | github.com/cockroachdb/pebble/internal/base.(*LazyValue).Value
| | external/com_github_cockroachdb_pebble/internal/base/lazy_value.go:195
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).getFromLazyValue
| | pkg/storage/pebble_mvcc_scanner.go:1595
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).decodeCurrentMetadata
| | pkg/storage/pebble_mvcc_scanner.go:1631
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).getOne
| | pkg/storage/pebble_mvcc_scanner.go:935
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).get
| | pkg/storage/pebble_mvcc_scanner.go:611
| | github.com/cockroachdb/cockroach/pkg/storage.mvccGet
| | pkg/storage/mvcc.go:1575
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGetWithValueHeader
| | pkg/storage/mvcc.go:1504
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGet
| | pkg/storage/mvcc.go:1404
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGetProto
| | pkg/storage/mvcc.go:1097
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvstorage.StateLoader.LoadLease
| | pkg/kv/kvserver/kvstorage/stateloader.go:142
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvstorage.StateLoader.Load
| | pkg/kv/kvserver/kvstorage/stateloader.go:57
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked
| | pkg/kv/kvserver/replica.go:2018
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | pkg/kv/kvserver/replica_raft.go:1296
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | pkg/kv/kvserver/replica_raft.go:884
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | pkg/kv/kvserver/store_raft.go:760
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | pkg/kv/kvserver/scheduler.go:421
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | pkg/kv/kvserver/scheduler.go:314
| | runtime.goexit
| | src/runtime/asm_arm64.s:1268
| Wraps: (4) open ×
| Wraps: (5) operation not permitted
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *fs.PathError (5) syscall.Errno
Wraps: (3) log.Dev.Fatal: pebble: backing file 000093 error: open ×: operation not permitted
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Dev.Fatal: pebble: backing file 000093 error: open ×: operation not permitted
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
replica.go:2020: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1267-L1269](src/runtime/asm_arm64.s#L1267-L1269)
[pkg/kv/kvserver/scheduler.go#L313-L315](pkg/kv/kvserver/scheduler.go#L313-L315)
[pkg/kv/kvserver/scheduler.go#L420-L422](pkg/kv/kvserver/scheduler.go#L420-L422)
[pkg/kv/kvserver/store_raft.go#L759-L761](pkg/kv/kvserver/store_raft.go#L759-L761)
[pkg/kv/kvserver/replica_raft.go#L883-L885](pkg/kv/kvserver/replica_raft.go#L883-L885)
[pkg/kv/kvserver/replica_raft.go#L1295-L1297](pkg/kv/kvserver/replica_raft.go#L1295-L1297)
[pkg/kv/kvserver/replica.go#L2019-L2021](pkg/kv/kvserver/replica.go#L2019-L2021)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1268
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 314
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 421
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 760
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 884
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1296
pkg/kv/kvserver/replica.go in pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked at line 2020
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v26.2.2 |
| Go Version | go1.25.5 |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v26.2.2 |
| Cockroach SHA | 1a79ea7f3f8dfffbc9f0f32dc05ecc1c6627d5bd |
| # of CPUs | 12 |
| # of Goroutines | 588 |

Jira issue: CRDB-64843

Start by reading pkg/kv/kvserver/replica.go around line 2020 and trace the reported path through replica_raft.go, pkg/storage/pebble_mvcc_scanner.go, and pkg/storage/fs/fs.go. Compare the failure with the supplied v26.2.2, Linux arm64, and start-single-node environment, then establish a reproducible case and add regression coverage or a documented fix once the cause is identified.

Written by the indexing model from the issue text.