Sentry: replica_application_result.go:469: log.Fatal: failed to update store after merging range: open ×: operation not permitted (1) attached stack trace -- stack trace: | github.com/cockroachdb...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/6893219708/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6893219708/?referrer=webhooks_plugin)

Panic Message:

```
replica_application_result.go:469: log.Fatal: failed to update store after merging range: open ×: operation not permitted
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleMergeResult
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_result.go:469
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:348
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:194
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:214
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1098
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:757
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
| runtime.goexit
| src/runtime/asm_arm64.s:1222
Wraps: (2) secondary error attachment
| open ×: operation not permitted
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/vfs.defaultFS.RemoveAll
| | github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/vfs.go:258
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).RemoveAll.func1
| | github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:845
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).timeFilesystemOp
| | github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:638
| | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).RemoveAll
| | github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:844
| | github.com/cockroachdb/pebble/vfs.(*enospcFS).RemoveAll
| | github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_full.go:268
| | github.com/cockroachdb/cockroach/pkg/storage/fs.(*Env).RemoveAll
| | github.com/cockroachdb/cockroach/pkg/storage/fs/fs.go:373
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*DiskSideloadStorage).Clear
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/sideload_disk.go:189
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).postDestroyRaftMuLocked
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_destroy.go:92
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).MergeRange
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_merge.go:144
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleMergeResult
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_result.go:459
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:348
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:194
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:214
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1098
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:757
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
| | runtime.goexit
| | src/runtime/asm_arm64.s:1222
| Wraps: (2) open ×
| Wraps: (3) operation not permitted
| Error types: (1) *withstack.withStack (2) *fs.PathError (3) syscall.Errno
Wraps: (3) log.Fatal: failed to update store after merging range: open ×: operation not permitted
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: failed to update store after merging range: open ×: operation not permitted
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
replica_application_result.go:469: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1221-L1223](src/runtime/asm_arm64.s#L1221-L1223)
[https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/util/stop/stopper.go#L484-L486](https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/util/stop/stopper.go#L484-L486)
[https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L320-L322](https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L320-L322)
[https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L417-L419](https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L417-L419)
[https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L688-L690](https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L688-L690)
[https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L756-L758](https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L756-L758)
[https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1097-L1099](https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1097-L1099)
[https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/apply/task.go#L250-L252](https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/apply/task.go#L250-L252)
[https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/apply/task.go#L294-L296](https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/apply/task.go#L294-L296)
[https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/apply/cmd.go#L213-L215](https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/apply/cmd.go#L213-L215)
[https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go#L193-L195](https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go#L193-L195)
[https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go#L347-L349](https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go#L347-L349)
[https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_result.go#L468-L470](https://github.com/cockroachdb/cockroach/blob/484f8e2f4d3cb23582416944d79eab72ff42a6a1/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_result.go#L468-L470)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1222
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 485
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 321
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 418
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 689
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 757
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1098
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries at line 251
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).applyOneBatch at line 295
pkg/kv/kvserver/apply/cmd.go in pkg/kv/kvserver/apply.mapCheckedCmdIter at line 214
pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go in pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects at line 194
pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go in pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult at line 348
pkg/kv/kvserver/pkg/kv/kvserver/replica_application_result.go in pkg/kv/kvserver.(*Replica).handleMergeResult at line 469
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v24.1.1 |
| Go Version | go1.22.3 X:nocoverageredesign |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v24.1.1 |
| Cockroach SHA | 484f8e2f4d3cb23582416944d79eab72ff42a6a1 |
| # of CPUs | 8 |
| # of Goroutines | 494 |

Jira issue: CRDB-54633