basalt-26.3.1-rc: kv/kvserver: TestCatchUpSnapshotSwapsRSManifest failed

kv/kvserver.TestCatchUpSnapshotSwapsRSManifest [failed](https://mesolite.cluster.engflow.com/invocations/default/6794c952-c9cd-4d74-b955-d80520562956?testReportRun=1&testReportShard=41&testReportAttempt=1#targets-Ly9wa2cva3Yva3ZzZXJ2ZXI6a3ZzZXJ2ZXJfdGVzdA==) on basalt-26.3.1-rc @ [63bc2c6772a94c6d763293cd0c2ec06e359c0a36](https://github.com/cockroachdb/cockroach/commits/63bc2c6772a94c6d763293cd0c2ec06e359c0a36):

```
| pkg/kv/kvclient/kvcoord/dist_sender.go:2333 +0x925
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc023fe1508, {0xce47598, 0xc022dc4fa0}, 0xc0214aef20, {{0xc0258abe98, 0x1, 0x8}, {0xc0258abea8, 0x2, 0x8}}, ...)
| pkg/kv/kvclient/kvcoord/dist_sender.go:1879 +0xf87
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc023fe1508, {0xce47598, 0xc01fa3b720}, 0xc0214aef20)
| pkg/kv/kvclient/kvcoord/dist_sender.go:1384 +0x80e
| github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(0xc003e1e118, {0xce47598, 0xc01fa3b720}, 0xc0214aef20)
| pkg/kv/db.go:232 +0x1e2
| github.com/cockroachdb/cockroach/pkg/kv.SendWrappedWithAdmission({_, _}, {_, _}, {{0x0, 0x0}, 0x0, {0x0, 0x0}, {0x0, ...}, ...}, ...)
| pkg/kv/sender.go:498 +0x224
| github.com/cockroachdb/cockroach/pkg/kv.SendWrappedWith(...)
| pkg/kv/sender.go:483
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*ReplicaManifestCommitter).InstallNewManifest(0xc01d325008, 0x0, {{{0xc02406d548, 0xf}, 0x8}, {0xc019ac88e8, 0x1, 0x1}}, {0x0, 0x0})
| pkg/kv/kvserver/replica_manifest_committer.go:258 +0xc57
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleManifestCommitter).InstallNewManifest(0xc01952e730, 0x0, {{{0xc02406d548, 0xf}, 0x8}, {0xc019ac88e8, 0x1, 0x1}, 0x2}, {0x0, ...})
| pkg/storage/rs_engine_pebble.go:306 +0x173
| github.com/cockroachdb/pebble.(*rsManifestInstaller).Install(0xc01b5f9d10, {0xc02635e2d0, 0x0, {0x0, {0x0, 0x0, 0x0}, 0x0}, {0x0, 0x0, ...}, ...})
| external/com_github_cockroachdb_pebble/manifest_installer.go:301 +0xd6e
| github.com/cockroachdb/pebble.(*versionSet).UpdateVersionLocked(0xc01fc93b08, 0xc01e134598)
| external/com_github_cockroachdb_pebble/version_set.go:781 +0x1bd7
| github.com/cockroachdb/pebble.(*DB).ingestApply(0xc019f32608, {0xce47598, 0xc01fa3b720}, 0x2, {{0xc0204b9630, 0x1, 0x1}, {0x0, 0x0, 0x0}, ...}, ...)
| external/com_github_cockroachdb_pebble/ingest.go:2770 +0x60e
| github.com/cockroachdb/pebble.(*DB).ingest.func2(0xa)
| external/com_github_cockroachdb_pebble/ingest.go:2478 +0x82d
| github.com/cockroachdb/pebble.(*commitPipeline).AllocateSeqNum(0xc025672000, 0x1, 0xc01e135318, 0xc01e1352a0)
| external/com_github_cockroachdb_pebble/commit.go:434 +0x8fe
| github.com/cockroachdb/pebble.(*DB).ingest(_, {_, _}, {{{0xc025f4ce70, 0x1, 0x1}, {0x0, 0x0, 0x0}}, {0x0, ...}, ...})
| external/com_github_cockroachdb_pebble/ingest.go:2498 +0x15be
| github.com/cockroachdb/pebble.(*DB).IngestWithHandle(0xc019f32608, {0xce47598, 0xc01fa3b720}, {0xc01e1359d8, 0x1, 0xc01e135978?}, {0x0, 0x0})
| external/com_github_cockroachdb_pebble/ingest.go:1798 +0x4f6
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleRSEngine).ingest(0xc0040f4d20, {0xc0245fb370, 0x1, 0x0?}, {0x0, 0x0})
| pkg/storage/rs_engine_pebble.go:116 +0x35b
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleRSEngine).flushSSTables(0xc0040f4d20, {0xc0245fb370, 0x1, 0x1}, 0x0)
| pkg/storage/rs_engine_pebble.go:102 +0x109
| github.com/cockroachdb/cockroach/pkg/storage.(*rsEngineContainer).FlushSSTables(0xc01aeb36c0, {0xc0245fb370, 0x1, 0x1}, 0x0)
| pkg/storage/rs_engine_container.go:241 +0x115
| github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.flushSSTIntoRSEngineE(0xc01d325008, {0xcef85d8, 0xc001eac200}, {0xc026c72000, 0x40b, 0x700})
| pkg/kv/kvserver/client_basalt_manifest_helpers_test.go:224 +0x310
| github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.testCatchUpSnapshotSwapsRSManifest(0xc01b61e008, {0xce20b40, 0x14e16e40})
| pkg/kv/kvserver/client_snapshot_manifest_test.go:239 +0x44b
| github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestCatchUpSnapshotSwapsRSManifest.func1(0xc01b61e008, {0xce20b40, 0x14e16e40})
| pkg/kv/kvserver/client_snapshot_manifest_test.go:184 +0x3b
| github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.runWithRSEngineImpls.func1(0xc01b61e008)
| pkg/kv/kvserver/client_basalt_manifest_helpers_test.go:71 +0x54
| testing.tRunner(0xc01b61e008, 0xc015cd0000)
| GOROOT/src/testing/testing.go:2036 +0x21d
| created by testing.(*T).Run in goroutine 25388
| GOROOT/src/testing/testing.go:2101 +0xb13
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errbase.opaqueWrapper (4) *errutil.withPrefix (5) *errbase.opaqueWrapper (6) *errutil.leafError
Test: TestCatchUpSnapshotSwapsRSManifest/pebble
--- FAIL: TestCatchUpSnapshotSwapsRSManifest/pebble (11.22s)
```

Parameters:
- attempt=1
- race=true
- run=1
- shard=41
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestCatchUpSnapshotSwapsRSManifest.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67207

Run kv/kvserver.TestCatchUpSnapshotSwapsRSManifest with the pebble implementation and inspect the failure around pkg/kv/kvserver/client_snapshot_manifest_test.go:239. Trace the stack through client_basalt_manifest_helpers_test.go, pkg/storage/rs_engine_pebble.go, and pkg/kv/kvserver/replica_manifest_committer.go. Done means the reported failure is understood and the test passes reliably.

Written by the indexing model from the issue text.