Sentry: replica_proposal.go:402: log.Fatal: failed checking for in-progress merge while installing new lease repl=(n5,s5):2 seq=49 start=1762014831.765229000,0 exp=1762014837.765109000,0 pro=176201483...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/6989863552/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6989863552/?referrer=webhooks_plugin)

Panic Message:

```
replica_proposal.go:402: log.Fatal: failed checking for in-progress merge while installing new lease repl=(n5,s5):2 seq=49 start=1762014831.765229000,0 exp=1762014837.765109000,0 pro=1762014831.765109000,0 acq=Transfer: pebble: backing file 008337 error: open ×: operation not permitted
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).leasePostApplyLocked
| pkg/kv/kvserver/replica_proposal.go:402
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleLeaseResult
| pkg/kv/kvserver/replica_application_result.go:495
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult
| pkg/kv/kvserver/replica_application_state_machine.go:291
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects
| pkg/kv/kvserver/replica_application_state_machine.go:198
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter
| pkg/kv/kvserver/apply/cmd.go:209
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| pkg/kv/kvserver/apply/task.go:290
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| pkg/kv/kvserver/apply/task.go:246
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1234
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:869
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:761
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:411
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:302
| runtime.goexit
| src/runtime/asm_arm64.s:1223
Wraps: (2) secondary error attachment
| pebble: backing file 008337 error: open ×: operation not permitted
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:468
| | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| | external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:191
| | github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
| | external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:84
| | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateTable
| | external/com_github_cockroachdb_pebble/file_cache.go:260
| | github.com/cockroachdb/pebble.(*fileCacheHandle).newIters
| | external/com_github_cockroachdb_pebble/file_cache.go:553
| | github.com/cockroachdb/pebble.(*levelIter).loadFile
| | external/com_github_cockroachdb_pebble/level_iter.go:568
| | github.com/cockroachdb/pebble.(*levelIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/level_iter.go:661
| | github.com/cockroachdb/pebble.(*mergingIter).seekGE
| | external/com_github_cockroachdb_pebble/merging_iter.go:969
| | github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGEStrict
| | external/com_github_cockroachdb_pebble/merging_iter.go:1044
| | github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/merging_iter.go:1035
| | github.com/cockroachdb/pebble.(*lazyCombinedIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/range_keys.go:618
| | github.com/cockroachdb/pebble.(*Iterator).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/iterator.go:1502
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
| | pkg/storage/pebble_iterator.go:343
| | github.com/cockroachdb/cockroach/pkg/storage.(*intentInterleavingIter).SeekGE
| | pkg/storage/intent_interleaving_iter.go:547
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).get
| | pkg/storage/pebble_mvcc_scanner.go:584
| | github.com/cockroachdb/cockroach/pkg/storage.mvccGet
| | pkg/storage/mvcc.go:1583
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGetWithValueHeader
| | pkg/storage/mvcc.go:1508
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGet
| | pkg/storage/mvcc.go:1408
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).maybeWatchForMergeLocked
| | pkg/kv/kvserver/replica.go:2521
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).leasePostApplyLocked
| | pkg/kv/kvserver/replica_proposal.go:399
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleLeaseResult
| | pkg/kv/kvserver/replica_application_result.go:495
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult
| | pkg/kv/kvserver/replica_application_state_machine.go:291
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects
| | pkg/kv/kvserver/replica_application_state_machine.go:198
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter
| | pkg/kv/kvserver/apply/cmd.go:209
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| | pkg/kv/kvserver/apply/task.go:290
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| | pkg/kv/kvserver/apply/task.go:246
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | pkg/kv/kvserver/replica_raft.go:1234
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | pkg/kv/kvserver/replica_raft.go:869
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | pkg/kv/kvserver/store_raft.go:761
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | pkg/kv/kvserver/scheduler.go:411
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | pkg/kv/kvserver/scheduler.go:302
| Wraps: (2) pebble: backing file 008337 error
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/vfs.defaultFS.Open
| | external/com_github_cockroachdb_pebble/vfs/vfs.go:275
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Open
| | external/com_github_cockroachdb_pebble/vfs/disk_health.go:805
| | github.com/cockroachdb/pebble/vfs.(*enospcFS).Open
| | external/com_github_cockroachdb_pebble/vfs/disk_full.go:221
| | github.com/cockroachdb/cockroach/pkg/storage/fs.(*Env).Open
| | pkg/storage/fs/fs.go:363
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).vfsOpenForReading
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs.go:27
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).OpenForReading
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:304
| | github.com/cockroachdb/pebble.(*fileCacheHandle).openFile
| | external/com_github_cockroachdb_pebble/file_cache.go:209
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:466
| | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| | external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:191
| | github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
| | external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:84
| | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateTable
| | external/com_github_cockroachdb_pebble/file_cache.go:260
| | github.com/cockroachdb/pebble.(*fileCacheHandle).newIters
| | external/com_github_cockroachdb_pebble/file_cache.go:553
| | github.com/cockroachdb/pebble.(*levelIter).loadFile
| | external/co...
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1222-L1224](src/runtime/asm_arm64.s#L1222-L1224)
[pkg/kv/kvserver/scheduler.go#L301-L303](pkg/kv/kvserver/scheduler.go#L301-L303)
[pkg/kv/kvserver/scheduler.go#L410-L412](pkg/kv/kvserver/scheduler.go#L410-L412)
[pkg/kv/kvserver/store_raft.go#L760-L762](pkg/kv/kvserver/store_raft.go#L760-L762)
[pkg/kv/kvserver/replica_raft.go#L868-L870](pkg/kv/kvserver/replica_raft.go#L868-L870)
[pkg/kv/kvserver/replica_raft.go#L1233-L1235](pkg/kv/kvserver/replica_raft.go#L1233-L1235)
[pkg/kv/kvserver/apply/task.go#L245-L247](pkg/kv/kvserver/apply/task.go#L245-L247)
[pkg/kv/kvserver/apply/task.go#L289-L291](pkg/kv/kvserver/apply/task.go#L289-L291)
[pkg/kv/kvserver/apply/cmd.go#L208-L210](pkg/kv/kvserver/apply/cmd.go#L208-L210)
[pkg/kv/kvserver/replica_application_state_machine.go#L197-L199](pkg/kv/kvserver/replica_application_state_machine.go#L197-L199)
[pkg/kv/kvserver/replica_application_state_machine.go#L290-L292](pkg/kv/kvserver/replica_application_state_machine.go#L290-L292)
[pkg/kv/kvserver/replica_application_result.go#L494-L496](pkg/kv/kvserver/replica_application_result.go#L494-L496)
[pkg/kv/kvserver/replica_proposal.go#L401-L403](pkg/kv/kvserver/replica_proposal.go#L401-L403)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 411
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 761
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 869
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1234
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries at line 246
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).applyOneBatch at line 290
pkg/kv/kvserver/apply/cmd.go in pkg/kv/kvserver/apply.mapCheckedCmdIter at line 209
pkg/kv/kvserver/replica_application_state_machine.go in pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects at line 198
pkg/kv/kvserver/replica_application_state_machine.go in pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult at line 291
pkg/kv/kvserver/replica_application_result.go in pkg/kv/kvserver.(*Replica).handleLeaseResult at line 495
pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).leasePostApplyLocked at line 402
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.3.2 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | darwin arm64 |
| Distribution | CCL |
| Cockroach Release | v25.3.2 |
| Cockroach SHA | a5e060ee11ad11e1649e7569486b579e9da8c8d7 |
| # of CPUs | 10 |
| # of Goroutines | 652 |

Jira issue: CRDB-56140

Start at pkg/kv/kvserver/replica_proposal.go:402 and trace the leasePostApplyLocked to maybeWatchForMergeLocked path into pkg/storage/mvcc.go and the reported Pebble file-open stack. Investigate why backing file 008337 returns operation not permitted on the reported environment, then establish a reproducible failure case and verify the crash is no longer triggered.

Written by the indexing model from the issue text.