basalt-26.3.1-rc: kv/kvserver: TestAdminMergeManifestDataDriven failed

kv/kvserver.TestAdminMergeManifestDataDriven [failed](https://mesolite.cluster.engflow.com/invocations/default/6794c952-c9cd-4d74-b955-d80520562956?testReportRun=1&testReportShard=25&testReportAttempt=1#targets-Ly9wa2cva3Yva3ZzZXJ2ZXI6a3ZzZXJ2ZXJfdGVzdA==) on basalt-26.3.1-rc @ [63bc2c6772a94c6d763293cd0c2ec06e359c0a36](https://github.com/cockroachdb/cockroach/commits/63bc2c6772a94c6d763293cd0c2ec06e359c0a36):

```
| github.com/cockroachdb/cockroach/pkg/kv.SendWrappedWith(...)
| pkg/kv/sender.go:483
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*ReplicaManifestCommitter).InstallNewManifest(0xc02778b008, 0x0, {{{0xc02aacdf71, 0xf}, 0x42}, {0xc021fb7500, 0x1, 0x1}}, {0x0, 0x0})
| pkg/kv/kvserver/replica_manifest_committer.go:258 +0xc57
| github.com/cockroachdb/cockroach/pkg/storage.(*TestingRSEngine).flushSSTables(0xc0294c6900, {0xc022e68560, 0x1, 0xc001867008?}, 0x0)
| pkg/storage/rs_engine_testing.go:431 +0xf12
| github.com/cockroachdb/cockroach/pkg/storage.(*rsEngineContainer).FlushSSTables(0xc01d15e9c0, {0xc022e68560, 0x1, 0x1}, 0x0)
| pkg/storage/rs_engine_container.go:241 +0x115
| github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.installTestManifestForKey(0xc0258ae008, 0xc0077dd808, {0xcef85d8, 0xc017fe4080}, {0xc02a2dfa48, 0x1, 0x8})
| pkg/kv/kvserver/client_basalt_manifest_helpers_test.go:159 +0x844
| github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.(*mergeManifestTestEnv).flushRange(...)
| pkg/kv/kvserver/client_merge_manifest_test.go:128
| github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.(*mergeManifestDDEnv).handleCommand(0xc0284f5880, {0xce46090, 0x14e16e40}, 0xc0258ae008, 0xc0035e6808)
| pkg/kv/kvserver/client_merge_manifest_test.go:284 +0x53e
| github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.testAdminMergeManifestDataDriven.func1.4(0xc0258ae008, 0xc0035e6808)
| pkg/kv/kvserver/client_merge_manifest_test.go:540 +0x19a
| github.com/cockroachdb/datadriven.RunTest.func1({0xcf03a58, 0xc0258ae008}, 0xc0035e6808)
| external/com_github_cockroachdb_datadriven/datadriven.go:120 +0x59
| github.com/cockroachdb/datadriven.runDirective.func1({0xcf03a58, 0xc0258ae008}, 0xc0035e6808, 0xc022e683d0)
| external/com_github_cockroachdb_datadriven/datadriven.go:367 +0x2b3
| github.com/cockroachdb/datadriven.runDirective({0xcf03a58, 0xc0258ae008}, 0xc0035e67e0, 0xc022e683d0)
| external/com_github_cockroachdb_datadriven/datadriven.go:373 +0x95
| github.com/cockroachdb/datadriven.runDirectiveOrSubTest({0xcf03a58, 0xc0258ae008}, 0xc0035e67e0, {0x0, 0x0}, 0xc022e683d0)
| external/com_github_cockroachdb_datadriven/datadriven.go:215 +0xa5
| github.com/cockroachdb/datadriven.runTestInternal({0xcf03a58, 0xc0258ae008}, {0xc002f680f0, 0xe2}, {0xcdd8780, 0xc01834b670}, 0xc022e683d0, 0x0)
| external/com_github_cockroachdb_datadriven/datadriven.go:188 +0xe5
| github.com/cockroachdb/datadriven.RunTestAny({0xcf03a58, 0xc0258ae008}, {0xc002f680f0, 0xe2}, 0xc022e683d0)
| external/com_github_cockroachdb_datadriven/datadriven.go:148 +0x3dc
| github.com/cockroachdb/datadriven.RunTest(0xc0258ae008, {0xc002f680f0, 0xe2}, 0xc018cee400)
| external/com_github_cockroachdb_datadriven/datadriven.go:119 +0xb8
| github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.testAdminMergeManifestDataDriven.func1(0xc0258ae008, {0xc002f680f0, 0xe2})
| pkg/kv/kvserver/client_merge_manifest_test.go:529 +0x8d0
| github.com/cockroachdb/datadriven.Walk.func1({0xcf03a58, 0xc0258ae008}, {0xc002f680f0, 0xe2})
| external/com_github_cockroachdb_datadriven/datadriven.go:451 +0x5f
| github.com/cockroachdb/datadriven.WalkAny({0xcf03a58, 0xc0258ae008}, {0xc002f680f0, 0xe2}, 0xc0028ec830)
| external/com_github_cockroachdb_datadriven/datadriven.go:462 +0x27b
| github.com/cockroachdb/datadriven.WalkAny.func1({0xcf03a58, 0xc0258ae008})
| external/com_github_cockroachdb_datadriven/datadriven.go:475 +0xfb
| github.com/cockroachdb/datadriven.subTest.func1(0xc0258ae008)
| external/com_github_cockroachdb_datadriven/test_interface.go:27 +0x3c
| testing.tRunner(0xc0258ae008, 0xc0013ea060)
| GOROOT/src/testing/testing.go:2036 +0x21d
| created by testing.(*T).Run in goroutine 44
| GOROOT/src/testing/testing.go:2101 +0xb13
Error types: (1) *errbase.opaqueWrapper (2) *errutil.withPrefix (3) *errbase.opaqueWrapper (4) *errutil.leafError
Test: TestAdminMergeManifestDataDriven/testing/admin_merge_rhs_check_fail
datadriven.go:343:
fatal during /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/testdata/admin_merge_manifest/testing/admin_merge_rhs_check_fail:8:

--- FAIL: TestAdminMergeManifestDataDriven/testing/admin_merge_rhs_check_fail (7.95s)
```

Parameters:
- attempt=1
- race=true
- run=1
- shard=25
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestAdminMergeManifestDataDriven.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67203

Start with pkg/kv/kvserver/client_merge_manifest_test.go, especially TestAdminMergeManifestDataDriven and the testing/admin_merge_rhs_check_fail case in pkg/kv/kvserver/testdata/admin_merge_manifest. Trace the failure through installTestManifestForKey and ReplicaManifestCommitter.InstallNewManifest, then reproduce the reported test failure. Done means the failing data-driven case passes reliably without regressing the surrounding merge-manifest tests.

Written by the indexing model from the issue text.