basalt-26.3.2-rc-20260908: kv/kvserver/gc: TestEndToEndGC failed

kv/kvserver/gc.TestEndToEndGC [failed](https://mesolite.cluster.engflow.com/invocations/default/8b446a13-190b-4a1b-bd84-04f0081537ff?testReportRun=3&testReportShard=1&testReportAttempt=1#targets-Ly9wa2cva3Yva3ZzZXJ2ZXIvZ2M6Z2NfdGVzdA==) on basalt-26.3.2-rc-20260908 @ [807558e5e81666af257c811e925e4cae570abc28](https://github.com/cockroachdb/cockroach/commits/807558e5e81666af257c811e925e4cae570abc28):

Fatal error:

```
panic: pebble: blob reference value size 14093 is greater than the blob file's value size 5480

```
Stack:

```
goroutine 1909052 [running]:
github.com/cockroachdb/pebble/internal/manifest.MakeBlobReference(0x1104dcdcbd78?, 0x1104df2ab7a0?, 0x3?, 0x1104e70d71d0)
external/com_github_cockroachdb_pebble/internal/manifest/blob_metadata.go:60 +0x17f
github.com/cockroachdb/pebble/internal/manifest.(*BulkVersionEdit).Apply(0x1104c1e35d98, 0x1104dcdca008, 0x3e80)
external/com_github_cockroachdb_pebble/internal/manifest/version_edit.go:1512 +0xd2f
github.com/cockroachdb/pebble.(*rsManifestInstaller).prepareExternalManifest(0x3?, 0x1d06, 0x1104dcdca008)
external/com_github_cockroachdb_pebble/manifest_installer.go:830 +0x1ab1
github.com/cockroachdb/pebble.(*rsManifestInstaller).PrepareExternalManifest.func1(...)
external/com_github_cockroachdb_pebble/manifest_installer.go:400
github.com/cockroachdb/pebble.(*rsManifestInstaller).PrepareExternalManifest(0x1104e4036fc0, 0x1d06)
external/com_github_cockroachdb_pebble/manifest_installer.go:409 +0x194
github.com/cockroachdb/pebble.(*DB).PrepareExternalManifest(...)
external/com_github_cockroachdb_pebble/db.go:3593
github.com/cockroachdb/cockroach/pkg/storage.(*pebbleRSEngine).prepareExternalManifest(0x1104dd2c0008?, 0x1104c1e366a0?)
pkg/storage/rs_engine_pebble.go:146 +0x30
github.com/cockroachdb/cockroach/pkg/storage.(*rsEngineContainer).PrepareExternalManifest(0x1104dd2c0008, 0x1d06, 0x1)
pkg/storage/rs_engine_container.go:531 +0x113
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).ApplyToStateMachine(0x1104dc10c1a0, {0x69ea5d0, 0x1104e5b48d20})
pkg/kv/kvserver/replica_app_batch.go:667 +0x1b4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0x1104c1e37438, {0x69ea5d0, 0x1104e5b48d20}, {0x6a25570, 0x1104dc10c5d8})
pkg/kv/kvserver/apply/task.go:285 +0x153
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0x1104c1e37438, {0x69ea5d0, 0x1104e5b48d20})
pkg/kv/kvserver/apply/task.go:246 +0x92
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
pkg/kv/kvserver/replica_raft.go:1246 +0x1c85
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
pkg/kv/kvserver/replica_raft.go:881 +0x1d3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0x1104d2ac8008, 0x1104c20458f0?)
pkg/kv/kvserver/store_raft.go:766 +0x15b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0x1104e4a3ce00, {0x69ea5d0, 0x1104d26663c0}, {0x6a15720, 0x1104d2ac8008}, 0x1104ee9dad88)
pkg/kv/kvserver/scheduler.go:421 +0x222
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x69ea5d0, 0x1104d26663c0}, 0x1104d24cbd08?)
pkg/kv/kvserver/scheduler.go:314 +0xe9
created by github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start in goroutine 1903447
pkg/kv/kvserver/scheduler.go:329 +0x27e
```
Log preceding fatal error

```
=== RUN TestEndToEndGC
=== RUN TestEndToEndGC/rangeTombstones=false/clearRange=true
test_log_scope.go:172: test logs captured to: outputs.zip/logTestEndToEndGC_rangeTombstones=false_clearRange=true2811472666
test_log_scope.go:82: use -show-logs to present logs inline
test_server_shim.go:168: automatically injected an external process virtual cluster under test; see comment at top of test_server_shim.go for details.
```

Parameters:
- attempt=1
- deadlock=true
- run=3
- shard=1
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #174874 basalt-26.3: kv/kvserver/gc: TestEndToEndGC failed [C-test-failure O-robot T-kv branch-basalt-26.3 release-blocker]
- #174868 kv/kvserver/gc: TestEndToEndGC failed [C-test-failure O-robot T-kv branch-master release-blocker]
- #174651 kv/kvserver/gc: TestEndToEndGC failed [C-test-failure O-robot T-kv branch-master]
- #174123 basalt-26.3: kv/kvserver/gc: TestEndToEndGC failed [C-test-failure O-robot T-kv X-duplicate branch-basalt-26.3]
- #173809 release-26.3.1-rc: kv/kvserver/gc: TestEndToEndGC failed [C-test-failure O-robot T-kv X-duplicate branch-release-26.3.1-rc]

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestEndToEndGC.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Start with kv/kvserver/gc.TestEndToEndGC and the linked failure report, then trace the stack through pkg/storage/rs_engine_pebble.go, pkg/storage/rs_engine_container.go, and pkg/kv/kvserver/replica_app_batch.go. Compare the repeated failures in the related issues and reproduce the failing clearRange=true case. Done means the test completes without the Pebble blob-reference panic across the affected branches.

Written by the indexing model from the issue text.