Sentry: version_set.go:1055: log.Dev.Fatal: MANIFEST flush failed: reading AEAD response header: × (1) attached stack trace -- stack trace: | github.com/cockroachdb/pebble.(*versionSet).UpdateVer...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7731673067/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7731673067/?referrer=webhooks_plugin)

Panic Message:

```
version_set.go:1055: log.Dev.Fatal: MANIFEST flush failed: reading AEAD response header: ×
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/pebble.(*versionSet).UpdateVersionLocked
| external/com_github_cockroachdb_pebble/version_set.go:1055
| github.com/cockroachdb/pebble.(*DB).compact1
| external/com_github_cockroachdb_pebble/compaction.go:2583
| github.com/cockroachdb/pebble.(*tableCompaction).Execute
| external/com_github_cockroachdb_pebble/compaction.go:426
| github.com/cockroachdb/pebble.(*DB).compact.func1.1
| external/com_github_cockroachdb_pebble/compaction.go:2393
| github.com/cockroachdb/pebble.(*DB).compact.func1
| external/com_github_cockroachdb_pebble/compaction.go:2422
| runtime/pprof.Do
| GOROOT/src/runtime/pprof/runtime.go:57
| github.com/cockroachdb/pebble.(*DB).compact
| external/com_github_cockroachdb_pebble/compaction.go:2387
| github.com/cockroachdb/pebble.(*DB).runPickedCompaction.func1
| external/com_github_cockroachdb_pebble/compaction.go:2277
| runtime.goexit
| src/runtime/asm_arm64.s:1447
Wraps: (2) secondary error attachment
| MANIFEST flush failed: reading AEAD response header: ×
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/manifestmanager.writeRecord
| | external/com_github_cockroachdb_pebble/internal/manifestmanager/manifestmanager.go:1034
| | github.com/cockroachdb/pebble/internal/manifestmanager.(*Manager).writeAndSync
| | external/com_github_cockroachdb_pebble/internal/manifestmanager/manifestmanager.go:875
| | github.com/cockroachdb/pebble/internal/manifestmanager.(*Manager).Append
| | external/com_github_cockroachdb_pebble/internal/manifestmanager/manifestmanager.go:810
| | github.com/cockroachdb/pebble.(*manifestWriter).Write
| | external/com_github_cockroachdb_pebble/manifest_writer.go:330
| | github.com/cockroachdb/pebble.(*versionSet).UpdateVersionLocked.func2
| | external/com_github_cockroachdb_pebble/version_set.go:1027
| | github.com/cockroachdb/pebble.(*versionSet).UpdateVersionLocked
| | external/com_github_cockroachdb_pebble/version_set.go:1042
| | github.com/cockroachdb/pebble.(*DB).compact1
| | external/com_github_cockroachdb_pebble/compaction.go:2583
| | github.com/cockroachdb/pebble.(*tableCompaction).Execute
| | external/com_github_cockroachdb_pebble/compaction.go:426
| | github.com/cockroachdb/pebble.(*DB).compact.func1.1
| | external/com_github_cockroachdb_pebble/compaction.go:2393
| | github.com/cockroachdb/pebble.(*DB).compact.func1
| | external/com_github_cockroachdb_pebble/compaction.go:2422
| | runtime/pprof.Do
| | GOROOT/src/runtime/pprof/runtime.go:57
| | github.com/cockroachdb/pebble.(*DB).compact
| | external/com_github_cockroachdb_pebble/compaction.go:2387
| | [...repeated from below...]
| Wraps: (2) MANIFEST flush failed
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/basalt/basaltclient.(*BlobDataClient).doRequestAEAD
| | external/com_github_cockroachdb_basalt_basaltclient/blob_data.go:274
| | github.com/cockroachdb/basalt/basaltclient.(*BlobDataClient).doRequest
| | external/com_github_cockroachdb_basalt_basaltclient/blob_data.go:216
| | github.com/cockroachdb/basalt/basaltclient.(*BlobDataClient).Append
| | external/com_github_cockroachdb_basalt_basaltclient/blob_data.go:332
| | github.com/cockroachdb/basalt/basaltclient.(*quorumReplicaWorker).run
| | external/com_github_cockroachdb_basalt_basaltclient/quorum_writer.go:812
| | runtime.goexit
| | src/runtime/asm_arm64.s:1447
| Wraps: (4) reading AEAD response header
| Wraps: (5) ×
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
Wraps: (3) log.Dev.Fatal: MANIFEST flush failed: reading AEAD response header: ×
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Dev.Fatal: MANIFEST flush failed: reading AEAD response header: ×
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
version_set.go:1055: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1446-L1448](src/runtime/asm_arm64.s#L1446-L1448)
[external/com_github_cockroachdb_pebble/compaction.go#L2276-L2278](external/com_github_cockroachdb_pebble/compaction.go#L2276-L2278)
[external/com_github_cockroachdb_pebble/compaction.go#L2386-L2388](external/com_github_cockroachdb_pebble/compaction.go#L2386-L2388)
[GOROOT/src/runtime/pprof/runtime.go#L56-L58](GOROOT/src/runtime/pprof/runtime.go#L56-L58)
[external/com_github_cockroachdb_pebble/compaction.go#L2421-L2423](external/com_github_cockroachdb_pebble/compaction.go#L2421-L2423)
[external/com_github_cockroachdb_pebble/compaction.go#L2392-L2394](external/com_github_cockroachdb_pebble/compaction.go#L2392-L2394)
[external/com_github_cockroachdb_pebble/compaction.go#L425-L427](external/com_github_cockroachdb_pebble/compaction.go#L425-L427)
[external/com_github_cockroachdb_pebble/compaction.go#L2582-L2584](external/com_github_cockroachdb_pebble/compaction.go#L2582-L2584)
[external/com_github_cockroachdb_pebble/version_set.go#L1054-L1056](external/com_github_cockroachdb_pebble/version_set.go#L1054-L1056)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1447
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).runPickedCompaction.func1 at line 2277
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact at line 2387
GOROOT/src/runtime/pprof/runtime.go in runtime/pprof.Do at line 57
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact.func1 at line 2422
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact.func1.1 at line 2393
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*tableCompaction).Execute at line 426
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact1 at line 2583
external/com_github_cockroachdb_pebble/version_set.go in github.com/cockroachdb/pebble.(*versionSet).UpdateVersionLocked at line 1055
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v26.3.2 |
| Go Version | go1.26.6 |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v26.3.2-cloudonly.1-basalt |
| Cockroach SHA | 807558e5e81666af257c811e925e4cae570abc28 |
| # of CPUs | 4 |
| # of Goroutines | 530 |

Jira issue: CRDB-68391

Start with external/com_github_cockroachdb_pebble/version_set.go at line 1055 and trace the MANIFEST flush through internal/manifestmanager/manifestmanager.go:1034. Then inspect external/com_github_cockroachdb_basalt_basaltclient/blob_data.go:274 and the linked compaction frames to identify why the AEAD response header cannot be read. Done means the production crash is explained and prevented or otherwise resolved, with evidence covering the reported linux arm64 environment.

Written by the indexing model from the issue text.