basalt-26.3.1-rc: kv/kvserver: TestSplitTxnManifestCheckRetry failed

kv/kvserver.TestSplitTxnManifestCheckRetry [failed](https://mesolite.cluster.engflow.com/invocations/default/6794c952-c9cd-4d74-b955-d80520562956?testReportRun=1&testReportShard=47&testReportAttempt=1#targets-Ly9wa2cva3Yva3ZzZXJ2ZXI6a3ZzZXJ2ZXJfdGVzdA==) on basalt-26.3.1-rc @ [63bc2c6772a94c6d763293cd0c2ec06e359c0a36](https://github.com/cockroachdb/cockroach/commits/63bc2c6772a94c6d763293cd0c2ec06e359c0a36):

```
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:185 +0x4fd
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.getChainUnaryInvoker.func4({0xce47598, 0xc020aace60}, {0xc9bdecc, 0x21}, {0xc824380, 0xc017b2d1e0}, {0xc660880, 0xc01dd2d300}, 0x0, {0x0, ...})
| pkg/rpc/context.go:1002 +0x18d
| github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3({0xce47598, 0xc020aacdc0}, 0xc017b2d080)
| pkg/rpc/context.go:887 +0x467
| github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
| pkg/rpc/context.go:1010
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc02882cf60, {0xce47598, 0xc020aacdc0}, 0x50d991?, {0xcdff248, 0xc01fa47aa0}, 0xc017b2d080)
| pkg/kv/kvclient/kvcoord/transport.go:229 +0x3ac
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc02882cf60, {0xce47598, 0xc020aacdc0}, 0xc017b2d080)
| pkg/kv/kvclient/kvcoord/transport.go:191 +0x135
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext({{0xcea3f18?, 0xc02882cf60?}}, {0xce47598, 0xc020aacdc0}, 0xc017b2d080)
| pkg/kv/kvclient/kvcoord/transport_race.go:70 +0x2df
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc001acd508, {0xce47598, 0xc020aacdc0}, 0xc017b2cf20, {0xc0068850a0, 0xc029004900}, 0x0)
| pkg/kv/kvclient/kvcoord/dist_sender.go:2923 +0x2a2e
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc001acd508, {0xce47598, 0xc020aacdc0}, 0xc017b2cf20, {{0xc002714528, 0x1, 0x8}, {0xc002714548, 0x2, 0x8}}, ...)
| pkg/kv/kvclient/kvcoord/dist_sender.go:2333 +0x925
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc001acd508, {0xce47598, 0xc020aacdc0}, 0xc017b2cf20, {{0xc002714528, 0x1, 0x8}, {0xc002714548, 0x2, 0x8}}, ...)
| pkg/kv/kvclient/kvcoord/dist_sender.go:1879 +0xf87
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc001acd508, {0xce47598, 0xc027ceedc0}, 0xc017b2cf20)
| pkg/kv/kvclient/kvcoord/dist_sender.go:1384 +0x80e
| github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(0xc013ed0cc8, {0xce47598, 0xc027ceedc0}, 0xc017b2cf20)
| pkg/kv/db.go:232 +0x1e2
| github.com/cockroachdb/cockroach/pkg/kv.SendWrappedWithAdmission({_, _}, {_, _}, {{0x0, 0x0}, 0x0, {0x0, 0x0}, {0x0, ...}, ...}, ...)
| pkg/kv/sender.go:498 +0x224
| github.com/cockroachdb/cockroach/pkg/kv.SendWrappedWith(...)
| pkg/kv/sender.go:483
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*ReplicaManifestCommitter).InstallNewManifest(0xc014efb008, 0x0, {{{0xc0028a2221, 0xf}, 0x42}, {0xc01f9a50c8, 0x1, 0x1}}, {0x0, 0x0})
| pkg/kv/kvserver/replica_manifest_committer.go:258 +0xc57
| github.com/cockroachdb/cockroach/pkg/storage.(*TestingRSEngine).flushSSTables(0xc004364ea0, {0xc002811620, 0x1, 0xc0000f9808?}, 0x0)
| pkg/storage/rs_engine_testing.go:431 +0xf12
| github.com/cockroachdb/cockroach/pkg/storage.(*rsEngineContainer).FlushSSTables(0xc01b0df860, {0xc002811620, 0x1, 0x1}, 0x0)
| pkg/storage/rs_engine_container.go:241 +0x115
| github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.installTestManifestForKey(0xc02a7226c8, 0xc01e06e008, {0xcef85d8, 0xc026305000}, {0xc002120258, 0x1, 0x8})
| pkg/kv/kvserver/client_basalt_manifest_helpers_test.go:159 +0x844
| github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.(*splitManifestTestEnv).installTestManifest(...)
| pkg/kv/kvserver/client_split_manifest_test.go:58
| github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.setupSplitManifestTestWithKnobs({0xce46090, 0x14e16e40}, 0xc02a7226c8, 0xc005d82e48)
| pkg/kv/kvserver/client_split_manifest_test.go:51 +0xdf
| github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestSplitTxnManifestCheckRetry.func1(0xc02a7226c8, {0xce20b10, 0x14e16e40})
| pkg/kv/kvserver/client_split_manifest_test.go:537 +0x428
| github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.runWithRSEngineImpls.func1(0xc02a7226c8)
| pkg/kv/kvserver/client_basalt_manifest_helpers_test.go:71 +0x54
| testing.tRunner(0xc02a7226c8, 0xc0250e18f0)
| GOROOT/src/testing/testing.go:2036 +0x21d
| created by testing.(*T).Run in goroutine 35567
| GOROOT/src/testing/testing.go:2101 +0xb13
Error types: (1) *errbase.opaqueWrapper (2) *errutil.withPrefix (3) *errbase.opaqueWrapper (4) *errutil.leafError
Test: TestSplitTxnManifestCheckRetry/testing
--- FAIL: TestSplitTxnManifestCheckRetry/testing (7.76s)
```

Parameters:
- attempt=1
- race=true
- run=1
- shard=47
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #172857 kv/kvserver: TestSplitTxnManifestCheckRetry failed [A-testing C-bug C-test-failure O-robot P-3 T-kv branch-master]

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestSplitTxnManifestCheckRetry.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67209

Start with TestSplitTxnManifestCheckRetry in pkg/kv/kvserver/client_split_manifest_test.go, then read installTestManifest in client_basalt_manifest_helpers_test.go and the stack frames through replica_manifest_committer.go. Reproduce the race-enabled test and compare the related failure in #172857. Done means the underlying failure is understood and the test passes reliably.

Written by the indexing model from the issue text.