cockroachdb / cockroachdb/cockroach
Sentry: vfs.go:275: pebble: backing file 1374347 error: open ×: permission denied (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:791 | github.com/co...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7637826868/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7637826868/?referrer=webhooks_plugin)
Panic Message:
```
vfs.go:275: pebble: backing file 1374347 error: open ×: permission denied
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| github.com/cockroachdb/cockroach/pkg/raft.LogSnapshot.slice
| pkg/raft/log.go:601
| github.com/cockroachdb/cockroach/pkg/raft.LogSnapshot.Slice
| pkg/raft/log.go:572
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1074
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:875
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:755
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:419
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:319
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:500
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/pebble.NewFileCache.func1
| external/com_github_cockroachdb_pebble/file_cache.go:461
| github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:191
| github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
| external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:84
| github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateTable
| external/com_github_cockroachdb_pebble/file_cache.go:243
| github.com/cockroachdb/pebble.(*fileCacheHandle).newIters
| external/com_github_cockroachdb_pebble/file_cache.go:546
| github.com/cockroachdb/pebble.(*levelIter).loadFile
| external/com_github_cockroachdb_pebble/level_iter.go:568
| github.com/cockroachdb/pebble.(*levelIter).SeekGE
| external/com_github_cockroachdb_pebble/level_iter.go:635
| github.com/cockroachdb/pebble.(*mergingIter).seekGE
| external/com_github_cockroachdb_pebble/merging_iter.go:978
| github.com/cockroachdb/pebble.(*mergingIter).SeekGE
| external/com_github_cockroachdb_pebble/merging_iter.go:1026
| github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit
| external/com_github_cockroachdb_pebble/iterator.go:1350
| github.com/cockroachdb/pebble.(*Iterator).SeekGE
| external/com_github_cockroachdb_pebble/iterator.go:1254
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
| pkg/storage/pebble_iterator.go:345
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftlog.(*Iterator).SeekGE
| pkg/kv/kvserver/raftlog/iterator.go:116
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftlog.Visit
| pkg/kv/kvserver/raftlog/iterator.go:154
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.LoadEntries
| pkg/kv/kvserver/logstore/logstore.go:699
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaRaftMuLogSnap).entriesRaftMuLocked
| pkg/kv/kvserver/replica_raftlog.go:221
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaRaftMuLogSnap).Entries
| pkg/kv/kvserver/replica_raftlog.go:204
| github.com/cockroachdb/cockroach/pkg/raft.LogSnapshot.slice
| pkg/raft/log.go:595
| github.com/cockroachdb/cockroach/pkg/raft.LogSnapshot.Slice
| pkg/raft/log.go:572
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1074
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:875
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:755
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:419
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:319
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:500
Wraps: (3) pebble: backing file 1374347 error
Wraps: (4) attached stack trace
-- stack trace:
| github.com/cockroachdb/pebble/vfs.defaultFS.Open
| external/com_github_cockroachdb_pebble/vfs/vfs.go:275
| github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Open
| external/com_github_cockroachdb_pebble/vfs/disk_health.go:805
| github.com/cockroachdb/pebble/vfs.(*enospcFS).Open
| external/com_github_cockroachdb_pebble/vfs/disk_full.go:221
| github.com/cockroachdb/cockroach/pkg/storage/fs.(*Env).Open
| pkg/storage/fs/fs.go:363
| github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).vfsOpenForReading
| external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs.go:27
| github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).OpenForReading
| external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:304
| github.com/cockroachdb/pebble.(*fileCacheHandle).openFile
| external/com_github_cockroachdb_pebble/file_cache.go:196
| github.com/cockroachdb/pebble.NewFileCache.func1
| external/com_github_cockroachdb_pebble/file_cache.go:459
| github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:191
| github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
| external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:84
| github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateTable
| external/com_github_cockroachdb_pebble/file_cache.go:243
| github.com/cockroachdb/pebble.(*fileCacheHandle).newIters
| external/com_github_cockroachdb_pebble/file_cache.go:546
| github.com/cockroachdb/pebble.(*levelIter).loadFile
| external/com_github_cockroachdb_pebble/level_iter.go:568
| github.com/cockroachdb/pebble.(*levelIter).SeekGE
| external/com_github_cockroachdb_pebble/level_iter.go:635
| github.com/cockroachdb/pebble.(*mergingIter).seekGE
| external/com_github_cockroachdb_pebble/merging_iter.go:978
| github.com/cockroachdb/pebble.(*mergingIter).SeekGE
| external/com_github_cockroachdb_pebble/merging_iter.go:1026
| github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit
| external/com_github_cockroachdb_pebble/iterator.go:1350
| github.com/cockroachdb/pebble.(*Iterator).SeekGE
| external/com_github_cockroachdb_pebble/iterator.go:1254
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
| pkg/storage/pebble_iterator.go:345
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftlog.(*Iterator).SeekGE
| pkg/kv/kvserver/raftlog/iterator.go:116
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftlog.Visit
| pkg/kv/kvserver/raftlog/iterator.go:154
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.LoadEntries
| pkg/kv/kvserver/logstore/logstore.go:699
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaRaftMuLogSnap).entriesRaftMuLocked
| pkg/kv/kvserver/replica_raftlog.go:221
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaRaftMuLogSnap).Entries
| pkg/kv/kvserver/replica_raftlog.go:204
| github.com/cockroachdb/cockroach/pkg/raft.LogSnapshot.slice
| pkg/raft/log.go:595
| github.com/cockroachdb/cockroach/pkg/raft.LogSnapshot.Slice
| pkg/raft/log.go:572
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1074
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:875
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:755
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:419
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:319
| github.com/cockroachd...
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_arm64.s#L1222-L1224](src/runtime/asm_arm64.s#L1222-L1224)
[pkg/util/stop/stopper.go#L499-L501](pkg/util/stop/stopper.go#L499-L501)
[pkg/kv/kvserver/scheduler.go#L318-L320](pkg/kv/kvserver/scheduler.go#L318-L320)
[pkg/kv/kvserver/scheduler.go#L418-L420](pkg/kv/kvserver/scheduler.go#L418-L420)
[pkg/kv/kvserver/store_raft.go#L754-L756](pkg/kv/kvserver/store_raft.go#L754-L756)
[pkg/kv/kvserver/replica_raft.go#L874-L876](pkg/kv/kvserver/replica_raft.go#L874-L876)
[pkg/kv/kvserver/replica_raft.go#L1073-L1075](pkg/kv/kvserver/replica_raft.go#L1073-L1075)
[pkg/raft/log.go#L571-L573](pkg/raft/log.go#L571-L573)
[pkg/raft/log.go#L600-L602](pkg/raft/log.go#L600-L602)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)
[src/runtime/asm_arm64.s#L1222-L1224](src/runtime/asm_arm64.s#L1222-L1224)
[pkg/util/stop/stopper.go#L499-L501](pkg/util/stop/stopper.go#L499-L501)
[pkg/kv/kvserver/scheduler.go#L318-L320](pkg/kv/kvserver/scheduler.go#L318-L320)
[pkg/kv/kvserver/scheduler.go#L418-L420](pkg/kv/kvserver/scheduler.go#L418-L420)
[pkg/kv/kvserver/store_raft.go#L754-L756](pkg/kv/kvserver/store_raft.go#L754-L756)
[pkg/kv/kvserver/replica_raft.go#L874-L876](pkg/kv/kvserver/replica_raft.go#L874-L876)
[pkg/kv/kvserver/replica_raft.go#L1073-L1075](pkg/kv/kvserver/replica_raft.go#L1073-L1075)
[pkg/raft/log.go#L571-L573](pkg/raft/log.go#L571-L573)
[pkg/raft/log.go#L594-L596](pkg/raft/log.go#L594-L596)
[pkg/kv/kvserver/replica_raftlog.go#L203-L205](pkg/kv/kvserver/replica_raftlog.go#L203-L205)
[pkg/kv/kvserver/replica_raftlog.go#L220-L222](pkg/kv/kvserver/replica_raftlog.go#L220-L222)
[pkg/kv/kvserver/logstore/logstore.go#L698-L700](pkg/kv/kvserver/logstore/logstore.go#L698-L700)
[pkg/kv/kvserver/raftlog/iterator.go#L153-L155](pkg/kv/kvserver/raftlog/iterator.go#L153-L155)
[pkg/kv/kvserver/raftlog/iterator.go#L115-L117](pkg/kv/kvserver/raftlog/iterator.go#L115-L117)
[pkg/storage/pebble_iterator.go#L344-L346](pkg/storage/pebble_iterator.go#L344-L346)
[external/com_github_cockroachdb_pebble/iterator.go#L1253-L1255](external/com_github_cockroachdb_pebble/iterator.go#L1253-L1255)
[external/com_github_cockroachdb_pebble/iterator.go#L1349-L1351](external/com_github_cockroachdb_pebble/iterator.go#L1349-L1351)
[external/com_github_cockroachdb_pebble/merging_iter.go#L1025-L1027](external/com_github_cockroachdb_pebble/merging_iter.go#L1025-L1027)
[external/com_github_cockroachdb_pebble/merging_iter.go#L977-L979](external/com_github_cockroachdb_pebble/merging_iter.go#L977-L979)
[external/com_github_cockroachdb_pebble/level_iter.go#L634-L636](external/com_github_cockroachdb_pebble/level_iter.go#L634-L636)
[external/com_github_cockroachdb_pebble/level_iter.go#L567-L569](external/com_github_cockroachdb_pebble/level_iter.go#L567-L569)
[external/com_github_cockroachdb_pebble/file_cache.go#L545-L547](external/com_github_cockroachdb_pebble/file_cache.go#L545-L547)
[external/com_github_cockroachdb_pebble/file_cache.go#L242-L244](external/com_github_cockroachdb_pebble/file_cache.go#L242-L244)
[external/com_github_cockroachdb_pebble/internal/genericcache/cache.go#L83-L85](external/com_github_cockroachdb_pebble/internal/genericcache/cache.go#L83-L85)
[external/com_github_cockroachdb_pebble/internal/genericcache/shard.go#L190-L192](external/com_github_cockroachdb_pebble/internal/genericcache/shard.go#L190-L192)
[external/com_github_cockroachdb_pebble/file_cache.go#L460-L462](external/com_github_cockroachdb_pebble/file_cache.go#L460-L462)
[pkg/util/stop/stopper.go#L499-L501](pkg/util/stop/stopper.go#L499-L501)
[pkg/kv/kvserver/scheduler.go#L318-L320](pkg/kv/kvserver/scheduler.go#L318-L320)
[pkg/kv/kvserver/scheduler.go#L418-L420](pkg/kv/kvserver/scheduler.go#L418-L420)
[pkg/kv/kvserver/store_raft.go#L754-L756](pkg/kv/kvserver/store_raft.go#L754-L756)
[pkg/kv/kvserver/replica_raft.go#L874-L876](pkg/kv/kvserver/replica_raft.go#L874-L876)
[pkg/kv/kvserver/replica_raft.go#L1073-L1075](pkg/kv/kvserver/replica_raft.go#L1073-L1075)
[pkg/raft/log.go#L571-L573](pkg/raft/log.go#L571-L573)
[pkg/raft/log.go#L594-L596](pkg/raft/log.go#L594-L596)
[pkg/kv/kvserver/replica_raftlog.go#L203-L205](pkg/kv/kvserver/replica_raftlog.go#L203-L205)
[pkg/kv/kvserver/replica_raftlog.go#L220-L222](pkg/kv/kvserver/replica_raftlog.go#L220-L222)
[pkg/kv/kvserver/logstore/logstore.go#L698-L700](pkg/kv/kvserver/logstore/logstore.go#L698-L700)
[pkg/kv/kvserver/raftlog/iterator.go#L153-L155](pkg/kv/kvserver/raftlog/iterator.go#L153-L155)
[pkg/kv/kvserver/raftlog/iterator.go#L115-L117](pkg/kv/kvserver/raftlog/iterator.go#L115-L117)
[pkg/storage/pebble_iterator.go#L344-L346](pkg/storage/pebble_iterator.go#L344-L346)
[external/com_github_cockroachdb_pebble/iterator.go#L1253-L1255](external/com_github_cockroachdb_pebble/iterator.go#L1253-L1255)
[external/com_github_cockroachdb_pebble/iterator.go#L1349-L1351](external/com_github_cockroachdb_pebble/iterator.go#L1349-L1351)
[external/com_github_cockroachdb_pebble/merging_iter.go#L1025-L1027](external/com_github_cockroachdb_pebble/merging_iter.go#L1025-L1027)
[external/com_github_cockroachdb_pebble/merging_iter.go#L977-L979](external/com_github_cockroachdb_pebble/merging_iter.go#L977-L979)
[external/com_github_cockroachdb_pebble/level_iter.go#L634-L636](external/com_github_cockroachdb_pebble/level_iter.go#L634-L636)
[external/com_github_cockroachdb_pebble/level_iter.go#L567-L569](external/com_github_cockroachdb_pebble/level_iter.go#L567-L569)
[external/com_github_cockroachdb_pebble/file_cache.go#L545-L547](external/com_github_cockroachdb_pebble/file_cache.go#L545-L547)
[external/com_github_cockroachdb_pebble/file_cache.go#L242-L244](external/com_github_cockroachdb_pebble/file_cache.go#L242-L244)
[external/com_github_cockroachdb_pebble/internal/genericcache/cache.go#L83-L85](external/com_github_cockroachdb_pebble/internal/genericcache/cache.go#L83-L85)
[external/com_github_cockroachdb_pebble/internal/genericcache/shard.go#L190-L192](external/com_github_cockroachdb_pebble/internal/genericcache/shard.go#L190-L192)
[external/com_github_cockroachdb_pebble/file_cache.go#L458-L460](external/com_github_cockroachdb_pebble/file_cache.go#L458-L460)
[external/com_github_cockroachdb_pebble/file_cache.go#L195-L197](external/com_github_cockroachdb_pebble/file_cache.go#L195-L197)
[external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go#L303-L305](external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go#L303-L305)
[external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs.go#L26-L28](external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs.go#L26-L28)
[pkg/storage/fs/fs.go#L362-L364](pkg/storage/fs/fs.go#L362-L364)
[external/com_github_cockroachdb_pebble/vfs/disk_full.go#L220-L222](external/com_github_cockroachdb_pebble/vfs/disk_full.go#L220-L222)
[external/com_github_cockroachdb_pebble/vfs/disk_health.go#L804-L806](external/com_github_cockroachdb_pebble/vfs/disk_health.go#L804-L806)
[external/com_github_cockroachdb_pebble/vfs/vfs.go#L274-L276](external/com_github_cockroachdb_pebble/vfs/vfs.go#L274-L276)
```
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 500
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 319
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 419
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 755
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 875
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1074
pkg/raft/log.go in pkg/raft.LogSnapshot.Slice at line 572
pkg/raft/log.go in pkg/raft.LogSnapshot.slice at line 601
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 500
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 319
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 419
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 755
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 875
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1074
pkg/raft/log.go in pkg/raft.LogSnapshot.Slice at line 572
pkg/raft/log.go in pkg/raft.LogSnapshot.slice at line 595
pkg/kv/kvserver/replica_raftlog.go in pkg/kv/kvserver.(*replicaRaftMuLogSnap).Entries at line 204
pkg/kv/kvserver/replica_raftlog.go in pkg/kv/kvserver.(*replicaRaftMuLogSnap).entriesRaftMuLocked at line 221
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.LoadEntries at line 699
pkg/kv/kvserver/raftlog/iterator.go in pkg/kv/kvserver/raftlog.Visit at line 154
pkg/kv/kvserver/raftlog/iterator.go in pkg/kv/kvserver/raftlog.(*Iterator).SeekGE at line 116
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).SeekGE at line 345
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).SeekGE at line 1254
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit at line 1350
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).SeekGE at line 1026
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).seekGE at line 978
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).SeekGE at line 635
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).loadFile at line 568
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).newIters at line 546
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateTable at line 243
external/com_github_cockroachdb_pebble/internal/genericcache/cache.go in github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate at line 84
external/com_github_cockroachdb_pebble/internal/genericcache/shard.go in github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue at line 191
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.NewFileCache.func1 at line 461
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 500
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 319
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 419
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 755
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 875
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1074
pkg/raft/log.go in pkg/raft.LogSnapshot.Slice at line 572
pkg/raft/log.go in pkg/raft.LogSnapshot.slice at line 595
pkg/kv/kvserver/replica_raftlog.go in pkg/kv/kvserver.(*replicaRaftMuLogSnap).Entries at line 204
pkg/kv/kvserver/replica_raftlog.go in pkg/kv/kvserver.(*replicaRaftMuLogSnap).entriesRaftMuLocked at line 221
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.LoadEntries at line 699
pkg/kv/kvserver/raftlog/iterator.go in pkg/kv/kvserver/raftlog.Visit at line 154
pkg/kv/kvserver/raftlog/iterator.go in pkg/kv/kvserver/raftlog.(*Iterator).SeekGE at line 116
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).SeekGE at line 345
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).SeekGE at line 1254
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit at line 1350
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).SeekGE at line 1026
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).seekGE at line 978
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).SeekGE at line 635
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).loadFile at line 568
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).newIters at line 546
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateTable at line 243
external/com_github_cockroachdb_pebble/internal/genericcache/cache.go in github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate at line 84
external/com_github_cockroachdb_pebble/internal/genericcache/shard.go in github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue at line 191
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.NewFileCache.func1 at line 459
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).openFile at line 196
external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go in github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).OpenForReading at line 304
external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs.go in github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).vfsOpenForReading at line 27
pkg/storage/fs/fs.go in pkg/storage/fs.(*Env).Open at line 363
external/com_github_cockroachdb_pebble/vfs/disk_full.go in github.com/cockroachdb/pebble/vfs.(*enospcFS).Open at line 221
external/com_github_cockroachdb_pebble/vfs/disk_health.go in github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Open at line 805
external/com_github_cockroachdb_pebble/vfs/vfs.go in github.com/cockroachdb/pebble/vfs.defaultFS.Open at line 275
```
### Tags
| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.2.3 |
| Go Version | go1.23.7 X:nocoverageredesign |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v25.2.3 |
| Cockroach SHA | adfe5601ff6cb72c0f65090995978ca9ebbdc647 |
| # of CPUs | 2 |
| # of Goroutines | 1546 |
Jira issue: CRDB-66149
Contributor guide
Research direction
Start with the panic path in pkg/raft/log.go:572-601, then trace the callers in pkg/kv/kvserver/replica_raftlog.go and pkg/kv/kvserver/logstore/logstore.go. Check how the permission-denied error from Pebble reaches LogSnapshot.Slice and determine the expected handling; the issue is complete when this live-cluster crash has a confirmed fix and its behavior is covered by an appropriate test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100