Sentry: replica.go:2149: log.Dev.Fatal: all replicas failed: object not found (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).assertStateRaft...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7625245929/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7625245929/?referrer=webhooks_plugin)

Panic Message:

```
replica.go:2149: log.Dev.Fatal: all replicas failed: object not found
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked
| pkg/kv/kvserver/replica.go:2149
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1293
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:881
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:766
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:421
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:314
| runtime.goexit
| src/runtime/asm_amd64.s:1771
Wraps: (2) secondary error attachment
| all replicas failed: object not found
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/basalt/basaltfs.(*file).readAtFramed
| | external/com_github_cockroachdb_basalt_basaltfs/file.go:235
| | github.com/cockroachdb/basalt/basaltfs.(*file).ReadAt
| | external/com_github_cockroachdb_basalt_basaltfs/file.go:128
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*vfsReadHandle).ReadAt
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs_readable.go:164
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:511
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:467
| | github.com/cockroachdb/pebble/sstable/blob.(*FileReader).ReadValueBlock
| | external/com_github_cockroachdb_pebble/sstable/blob/blob.go:571
| | github.com/cockroachdb/pebble/sstable/blob.(*cachedReader).GetUnsafeValue
| | external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go:332
| | github.com/cockroachdb/pebble/sstable/blob.(*ValueFetcher).retrieve
| | external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go:211
| | github.com/cockroachdb/pebble/sstable/blob.(*ValueFetcher).FetchHandle
| | external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go:126
| | github.com/cockroachdb/pebble/internal/base.(*LazyValue).Value
| | external/com_github_cockroachdb_pebble/internal/base/lazy_value.go:195
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).getFromLazyValue
| | pkg/storage/pebble_mvcc_scanner.go:1595
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).decodeCurrentMetadata
| | pkg/storage/pebble_mvcc_scanner.go:1631
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).getOne
| | pkg/storage/pebble_mvcc_scanner.go:935
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).get
| | pkg/storage/pebble_mvcc_scanner.go:611
| | github.com/cockroachdb/cockroach/pkg/storage.mvccGet
| | pkg/storage/mvcc.go:1585
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGetWithValueHeader
| | pkg/storage/mvcc.go:1514
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGet
| | pkg/storage/mvcc.go:1414
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGetProto
| | pkg/storage/mvcc.go:1107
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvstorage.StateLoader.LoadLease
| | pkg/kv/kvserver/kvstorage/stateloader.go:152
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvstorage.StateLoader.Load
| | pkg/kv/kvserver/kvstorage/stateloader.go:57
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked
| | pkg/kv/kvserver/replica.go:2147
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | pkg/kv/kvserver/replica_raft.go:1293
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | pkg/kv/kvserver/replica_raft.go:881
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | pkg/kv/kvserver/store_raft.go:766
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | pkg/kv/kvserver/scheduler.go:421
| | [...repeated from below...]
| Wraps: (2) all replicas failed
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/basalt/basaltclient.init
| | external/com_github_cockroachdb_basalt_basaltclient/blob_protocol.go:211
| | runtime.doInit1
| | GOROOT/src/runtime/proc.go:8385
| | runtime.doInit
| | GOROOT/src/runtime/proc.go:8352
| | runtime.main
| | GOROOT/src/runtime/proc.go:258
| | runtime.goexit
| | src/runtime/asm_amd64.s:1771
| Wraps: (4) object not found
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
Wraps: (3) log.Dev.Fatal: all replicas failed: object not found
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Dev.Fatal: all replicas failed: object not found
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
replica.go:2149: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1770-L1772](src/runtime/asm_amd64.s#L1770-L1772)
[pkg/kv/kvserver/scheduler.go#L313-L315](pkg/kv/kvserver/scheduler.go#L313-L315)
[pkg/kv/kvserver/scheduler.go#L420-L422](pkg/kv/kvserver/scheduler.go#L420-L422)
[pkg/kv/kvserver/store_raft.go#L765-L767](pkg/kv/kvserver/store_raft.go#L765-L767)
[pkg/kv/kvserver/replica_raft.go#L880-L882](pkg/kv/kvserver/replica_raft.go#L880-L882)
[pkg/kv/kvserver/replica_raft.go#L1292-L1294](pkg/kv/kvserver/replica_raft.go#L1292-L1294)
[pkg/kv/kvserver/replica.go#L2148-L2150](pkg/kv/kvserver/replica.go#L2148-L2150)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1771
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 314
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 421
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 766
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 881
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1293
pkg/kv/kvserver/replica.go in pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked at line 2149
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v26.3.0-rc.1 |
| Go Version | go1.26.4 |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v26.3.0-rc.1-cloudonly.1-basalt |
| Cockroach SHA | 31b1b05e4ffc3f6152af230b2c2489a7b28587a3 |
| # of CPUs | 4 |
| # of Goroutines | 415 |

Jira issue: CRDB-65968

Start at pkg/kv/kvserver/replica.go:2149 and trace the failure through replica_raft.go:1293, kvstorage/stateloader.go:152, and the Pebble/Basalt read stack in pkg/storage. Use the Sentry report and linked stack frames to establish why object not found reaches the fatal path; done means the cause is identified and a verified resolution prevents this crash.

Written by the indexing model from the issue text.