kvclient: potential data race between kvserver and kvclient

**Describe the problem**

I'm experimenting with allowing KVNemesis under the race detector. Doing so revealed the following race-detector finding. I have not yet investigated this to understand if this is a real issue or a test-only issue:

```
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pkg/kv/kvnemesis:kvnemesis_test
-----------------------------------------------------------------------------
metamorphic: use COCKROACH_RANDOM_SEED=-5707873344410533378 for reproduction
initialized metamorphic constant "max-batch-size" with value 2955
initialized metamorphic constant "max-batch-byte-size" with value 15663944
initialized metamorphic constant "parse-json-impl" with value 1
initialized metamorphic constant "default-batch-bytes-limit" with value 53896
initialized metamorphic constant "kvadmission.flow_control.mode" with value apply_to_elastic
initialized metamorphic constant "kvadmission.store.snapshot_ingest_bandwidth_control.enabled" with value false
initialized metamorphic constant "spilling-queue-initial-len" with value 4
initialized metamorphic constant "merge-joiner-groups-buffer" with value 14
initialized metamorphic constant "distsender-leaseholder-first" with value false
initialized metamorphic constant "row-container-rows-per-chunk-shift" with value 1
initialized metamorphic constant "lock-table-iters-before-seek" with value 0
initialized metamorphic constant "storage.columnar_blocks.enabled" with value false
initialized metamorphic constant "storage.ingest_as_flushable.enabled" with value false
initialized metamorphic constant "disable-checksstconflicts-range-key-masking" with value true
initialized metamorphic constant "storage.ingestion.value_blocks.enabled" with value false
initialized metamorphic constant "raft-log-truncation-clearrange-threshold" with value 589621
initialized metamorphic constant "addsst-rewrite-concurrency" with value 12
initialized metamorphic constant "kv.snapshot.ingest_as_write_threshold" with value 102400
initialized metamorphic constant "kv.lease.expiration_leases_only.enabled" with value true
initialized metamorphic constant "kv.raft.leader_fortification.fraction_enabled" with value 1
initialized metamorphic constant "kv.snapshot_rebalance.max_sst_size" with value 20948462
initialized metamorphic constant "kv-batch-size" with value 1
initialized metamorphic constant "datum-row-converter-batch-size" with value 1
initialized metamorphic constant "direct-scans-enabled" with value true
initialized metamorphic constant "ColIndexJoin-using-streamer-batch-size" with value 7418298
initialized metamorphic constant "parallel-scan-result-threshold" with value 3339
initialized metamorphic constant "inverted-joiner-batch-size" with value 1
initialized metamorphic constant "copy-batch-size" with value 40080
initialized metamorphic constant "use-index-lookup-for-descriptors-in-database" with value false
initialized metamorphic constant "tail-call-optimization-enabled" with value false
I250402 09:43:10.808811 1 (gostd) rand.go:256 [-] 1 random seed: -5707873344410533378
=== RUN TestKVNemesisSingleNode
test_log_scope.go:165: test logs captured to: /tmp/cockroach/_tmp/3071ecfd607c2dcda710d9ffa8fdef40/logTestKVNemesisSingleNode97231295
test_log_scope.go:76: use -show-logs to present logs inline
kvnemesis_test.go:346: seed: 7056526346366980836
kvnemesis_test.go:235: kvnemesis logging to /tmp/cockroach/_tmp/3071ecfd607c2dcda710d9ffa8fdef40/kvnemesis3109831935
==================
WARNING: DATA RACE
Read at 0x00c0010fedd0 by goroutine 645:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).updateTimestampCache()
pkg/kv/kvserver/replica_tscache.go:288 +0xedc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*endCmds).done()
pkg/kv/kvserver/replica_send.go:1340 +0xf4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*ProposalData).finishApplication()
pkg/kv/kvserver/replica_proposal.go:257 +0x84
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicatedCmd).AckOutcomeAndFinish()
pkg/kv/kvserver/replica_application_cmd.go:151 +0xc0
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.AppliedCommand.AckOutcomeAndFinish()
:1 +0x50
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.forEachAppliedCmdIter()
pkg/kv/kvserver/apply/cmd.go:263 +0x108
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch()
pkg/kv/kvserver/apply/task.go:296 +0x234
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch()
pkg/kv/kvserver/apply/task.go:290 +0x1e0
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries()
pkg/kv/kvserver/apply/task.go:246 +0xa0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked()
pkg/kv/kvserver/replica_raft.go:1278 +0x1848
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady()
pkg/kv/kvserver/replica_raft.go:875 +0x158
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady()
pkg/kv/kvserver/store_raft.go:755 +0x148
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker()
pkg/kv/kvserver/scheduler.go:419 +0x294
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker()
pkg/kv/kvserver/scheduler.go:419 +0x294
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker()
pkg/kv/kvserver/scheduler.go:419 +0x294
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker()
pkg/kv/kvserver/scheduler.go:419 +0x294
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker()
pkg/kv/kvserver/scheduler.go:419 +0x294
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker()
pkg/kv/kvserver/scheduler.go:419 +0x294
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker()
pkg/kv/kvserver/scheduler.go:419 +0x294
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker()
pkg/kv/kvserver/scheduler.go:419 +0x294
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker()
pkg/kv/kvserver/scheduler.go:419 +0x294
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2()
pkg/kv/kvserver/scheduler.go:319 +0x94
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
pkg/util/stop/stopper.go:500 +0x214
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.gowrap1()
pkg/util/stop/stopper.go:501 +0x54

Previous write at 0x00c0010fedd0 by goroutine 2878:
github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*ResponseHeader).combine()
pkg/kv/kvpb/api.go:409 +0xb8
github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*ScanResponse).combine()
pkg/kv/kvpb/api.go:474 +0xb94
github.com/cockroachdb/cockroach/pkg/kv/kvpb.CombineResponses()
pkg/kv/kvpb/api.go:393 +0xb0
github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*BatchResponse).Combine()
pkg/kv/kvpb/batch.go:722 +0x338
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges.func1()
pkg/kv/kvclient/kvcoord/dist_sender.go:1857 +0x394
runtime.deferreturn()
GOROOT/src/runtime/panic.go:611 +0x5c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
pkg/kv/kvclient/kvcoord/dist_sender.go:1265 +0x788
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked()
pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:77 +0x1a0
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked()
pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:41 +0x16c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts()
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:234 +0x1c0
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked()
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:162 +0x160
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked()
pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:144 +0x6f4
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked()
pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:334 +0x1f8
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnWriteBuffer).SendLocked()
pkg/kv/kvclient/kvcoord/txn_interceptor_write_buffer.go:131 +0x1c8
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked()
pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:112 +0x2bc
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked()
pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:265 +0x524
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send()
pkg/kv/kvclient/kvcoord/txn_coord_sender.go:554 +0x784
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
pkg/kv/db.go:1166 +0xfc
... SNIP ...

```

[test.log](https://github.com/user-attachments/files/19565732/test.log)

Jira issue: CRDB-49064