Sentry: replica.go:2176: log.Dev.Fatal: all replicas failed: reading AEAD response header: × (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica)...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7740430910/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7740430910/?referrer=webhooks_plugin)

Panic Message:

```
replica.go:2176: log.Dev.Fatal: all replicas failed: reading AEAD response header: ×
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked
| pkg/kv/kvserver/replica.go:2176
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1294
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:881
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:766
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:421
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:314
| runtime.goexit
| src/runtime/asm_arm64.s:1447
Wraps: (2) secondary error attachment
| all replicas failed: reading AEAD response header: ×
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/basalt/basaltfs.(*file).readFromReplicas
| | external/com_github_cockroachdb_basalt_basaltfs/file.go:455
| | github.com/cockroachdb/basalt/basaltfs.(*file).readAtFramed
| | external/com_github_cockroachdb_basalt_basaltfs/file.go:586
| | github.com/cockroachdb/basalt/basaltfs.(*file).ReadAt
| | external/com_github_cockroachdb_basalt_basaltfs/file.go:355
| | github.com/cockroachdb/cockroach/pkg/storage/fs.(*encryptedFile).ReadAt
| | pkg/storage/fs/encryption_at_rest.go:520
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*fileReadable).ReadAt
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs_readable.go:84
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*fileReadable).readInternal
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs_readable.go:93
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*bufferedReadHandle).ReadAt
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/buffered_read_handle.go:196
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:612
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:480
| | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:431
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:514
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).seekGEHelper
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:836
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).seekPrefixGE
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1088
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1016
| | github.com/cockroachdb/pebble/internal/iterv2.(*InterleavingIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/internal/iterv2/interleaving_iter.go:464
| | github.com/cockroachdb/pebble.(*levelIterV2).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/level_iter_v2.go:546
| | github.com/cockroachdb/pebble.(*mergingIterV2).seekGE-range1
| | external/com_github_cockroachdb_pebble/merging_iter_v2.go:513
| | github.com/cockroachdb/pebble.(*mergingIterV2).seekGE.(*slabState).Build.func1
| | external/com_github_cockroachdb_pebble/merging_iter_v2_slab.go:154
| | github.com/cockroachdb/pebble.(*mergingIterV2).seekGE
| | external/com_github_cockroachdb_pebble/merging_iter_v2.go:475
| | github.com/cockroachdb/pebble.(*mergingIterV2).SeekPrefixGEStrict
| | external/com_github_cockroachdb_pebble/merging_iter_v2.go:976
| | github.com/cockroachdb/pebble.(*mergingIterV2).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/merging_iter_v2.go:935
| | github.com/cockroachdb/pebble.(*lazyCombinedIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/range_keys.go:626
| | github.com/cockroachdb/pebble.(*Iterator).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/iterator.go:1711
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
| | pkg/storage/pebble_iterator.go:369
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).get
| | pkg/storage/pebble_mvcc_scanner.go:585
| | github.com/cockroachdb/cockroach/pkg/storage.mvccGet
| | pkg/storage/mvcc.go:1582
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGetWithValueHeader
| | pkg/storage/mvcc.go:1511
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGet
| | pkg/storage/mvcc.go:1411
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCGetProto
| | pkg/storage/mvcc.go:1104
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.StateLoader.LoadRaftTruncatedState
| | pkg/kv/kvserver/logstore/stateloader.go:111
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked
| | pkg/kv/kvserver/replica.go:2175
| Wraps: (2) all replicas failed
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/basalt/basaltclient.(*BlobDataClient).doRequestAEAD
| | external/com_github_cockroachdb_basalt_basaltclient/blob_data.go:277
| | github.com/cockroachdb/basalt/basaltclient.(*BlobDataClient).doRequest
| | external/com_github_cockroachdb_basalt_basaltclient/blob_data.go:219
| | github.com/cockroachdb/basalt/basaltclient.(*BlobDataClient).Auth
| | external/com_github_cockroachdb_basalt_basaltclient/blob_data.go:759
| | github.com/cockroachdb/basalt/basaltfs.(*file).tryReplicas
| | external/com_github_cockroachdb_basalt_basaltfs/file.go:489
| | github.com/cockroachdb/basalt/basaltfs.(*file).readFromReplicas
| | external/com_github_cockroachdb_basalt_basaltfs/file.go:428
| | github.com/cockroachdb/basalt/basaltfs.(*file).readAtFramed
| | external/com_github_cockroachdb_basalt_basaltfs/file.go:586
| | github.com/cockroachdb/basalt/basaltfs.(*file).ReadAt
| | external/com_github_cockroachdb_basalt_basaltfs/file.go:355
| | github.com/cockroachdb/cockroach/pkg/storage/fs.(*encryptedFile).ReadAt
| | pkg/storage/fs/encryption_at_rest.go:520
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*fileReadable).ReadAt
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs_readable.go:84
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*fileReadable).readInternal
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs_readable.go:93
| | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*bufferedReadHandle).ReadAt
| | external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/buffered_read_handle.go:196
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:612
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:480
| | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:431
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:514
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).seekGEHelper
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:836
| | github.com/cockr...
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1446-L1448](src/runtime/asm_arm64.s#L1446-L1448)
[pkg/kv/kvserver/scheduler.go#L313-L315](pkg/kv/kvserver/scheduler.go#L313-L315)
[pkg/kv/kvserver/scheduler.go#L420-L422](pkg/kv/kvserver/scheduler.go#L420-L422)
[pkg/kv/kvserver/store_raft.go#L765-L767](pkg/kv/kvserver/store_raft.go#L765-L767)
[pkg/kv/kvserver/replica_raft.go#L880-L882](pkg/kv/kvserver/replica_raft.go#L880-L882)
[pkg/kv/kvserver/replica_raft.go#L1293-L1295](pkg/kv/kvserver/replica_raft.go#L1293-L1295)
[pkg/kv/kvserver/replica.go#L2175-L2177](pkg/kv/kvserver/replica.go#L2175-L2177)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1447
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 314
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 421
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 766
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 881
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1294
pkg/kv/kvserver/replica.go in pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked at line 2176
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v26.3.2 |
| Go Version | go1.26.6 |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v26.3.2-cloudonly.3-basalt |
| Cockroach SHA | 6afeecf4746420462fcd1245652a8933ddcfca2f |
| # of CPUs | 4 |
| # of Goroutines | 836 |

Jira issue: CRDB-68503

Start with pkg/kv/kvserver/replica.go at lines 2175-2177 and follow the reported path through replica_raft.go, store_raft.go, and pkg/storage/fs/encryption_at_rest.go. Inspect the Sentry report and the Basalt calls in the stack trace to determine the failure scope. Done should include an identified cause, a validated fix, and a regression test or reproduction for the AEAD response-header failure.

Written by the indexing model from the issue text.