live-lock (or close to) on single-threaded batch-insert of vectors into table with vector index

**Describe the problem**

I was conducting a benchmark on cockroach as I'm considering it for a project.

I had a single-threaded program running an insert into a table with a vector index declared on it. I could regularly get the database to enter a situation that was live-lock-like (progress would eventually be made, but it could take 30+ minutes). CPU and I/O were essentially nothing (as reported by `htop`).

The database would not report an error back to the driver, my program would just wait and wait.

The program would insert vectors in batches of 2000 (I'd send over 200 inserts as one SQL statement). After 30-40K inserts the database would stop accepting inserts and my program would seemingly hang. If I let 30 minutes or so elapse inserts would resume momentarily before hanging again for long periods of time (30+ minutes).

Note that I hadn't tested during this live-lock-like scenario to see if other queries would hang, but I suspect it's just the active transaction that is making very little progress.

**To Reproduce**

1. I setup a 3-node cockroach cluster on a single machine with 5TB of storage, 96 cores, and 500GB worth of memory. Each node ran in a docker container and got roughly an equal split of those resources (save a handful of cores and 25GB of memory).
2. Create the following table and index

```
CREATE TABLE IF NOT EXISTS datapoints (
id BIGINT PRIMARY KEY,
source_id SMALLINT NOT NULL REFERENCES sources (id),
text_embedding_v3 VECTOR(512)
);

CREATE VECTOR INDEX IF NOT EXISTS %s ON datapoints (source_id, text_embedding_v3 vector_cosine_ops) WITH (min_partition_size=1024, max_partition_size=4096);
```

3. Run a golang script that generates INSERT statements for the table in batches of 2000 and executes them. The goal was to insert 40M records total largely centered around 40K centroids.
4. The script will print out a statement for each batch it inserts. After 30-40k insertions I notice I stop getting updates, I/O is very low, and CPU usage is very low. If I wait 30 minutes some activity might happen, I might get in another few batches of 500, but then again largely nothing for large periods of time.

**Additional data / screenshots**
I looked at the stack traces for the go routines and I had 22 go routines (I had 28 cores allotted to each container) all with this stack trace:

```
goroutine 41849090 [select, 9 minutes]:
github.com/cockroachdb/cockroach/pkg/rpc.(*pooledStream[...]).Send(...)
pkg/rpc/stream_pool.go:169
github.com/cockroachdb/cockroach/pkg/rpc.(*streamPool[...]).Send(0x8e1e3c0, {0x8d5fe28, 0xc0a581aaa0}, 0xc0a58266e0)
pkg/rpc/stream_pool.go:333 +0x20e
github.com/cockroachdb/cockroach/pkg/rpc/nodedialer.(*batchStreamPoolClient).Batch(0x8d5fe28?, {0x8d5fe28?, 0xc0a581aaa0?}, 0x1?)
pkg/rpc/nodedialer/nodedialer.go:400 +0x2d
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc224295860, {0x8d5fe28, 0xc0a581aaa0}, 0xaf672808?, {0x8d20228, 0xc0167bb1c0}, 0xc0a58266e0)
pkg/kv/kvclient/kvcoord/transport.go:228 +0x575
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(...)
pkg/kv/kvclient/kvcoord/transport.go:191
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc011fba008, {0x8d5fe28, 0xc0a581aaa0}, 0xc0a5826580, {0xc0085f3f20, 0xc0a18fae00}, 0x0?)
pkg/kv/kvclient/kvcoord/dist_sender.go:2905 +0x2d25
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc011fba008, {0x8d5fe28, 0xc0a581aaa0}, 0xc0a5826580, {{0xc0b0b19280, 0x10, 0x20}, {0xc0b0b19280, 0x11, 0x20}}, ...)
pkg/kv/kvclient/kvcoord/dist_sender.go:2326 +0x6ca
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc011fba008, {0x8d5fe28, 0xc0a581aaa0}, 0xc0a5826580, {{0xc0b0b19280, 0x10, 0x20}, {0xc0b0b19280, 0x11, 0x20}}, ...)
pkg/kv/kvclient/kvcoord/dist_sender.go:1872 +0xaba
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc011fba008, {0x8d5dbf8, 0xc0a5a94640}, 0xc0a5826580)
pkg/kv/kvclient/kvcoord/dist_sender.go:1377 +0x13f0
github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(0xc011fb0278, {0x8d5dbf8, 0xc0a5a94640}, 0xc0a5826580)
pkg/kv/db.go:232 +0x122
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc011fb0210, {0x8d5dbf8, 0xc0a5a94640}, 0xc0a5826580, {0x8cf5120, 0xc011fb0278})
pkg/kv/db.go:1246 +0xe7
github.com/cockroachdb/cockroach/pkg/kv.(*DB).send(0xc011fb0210?, {0x8d5dbf8?, 0xc0a5a94640?}, 0xc0a5826580?)
pkg/kv/db.go:1229 +0xaf
github.com/cockroachdb/cockroach/pkg/kv.sendAndFill({0x8d5dbf8, 0xc0a5a94640}, 0xc1182515c0, 0xc0a2123808)
pkg/kv/db.go:1023 +0x130
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run(0xc011fb0210, {0x8d5dbf8, 0xc0a5a94640}, 0xc0a2123808)
pkg/kv/db.go:1046 +0x65
github.com/cockroachdb/cockroach/pkg/kv.(*DB).CPut(0xc011fb0210, {0x8d5dbf8, 0xc0a5a94640}, {0x78dec20, 0xc200c2b110}, {0x6c3fa80, 0xc200c2b128}, {0xc0a6462004, 0x82f, 0x82f})
pkg/kv/db.go:504 +0xb9
github.com/cockroachdb/cockroach/pkg/sql/vecindex/vecstore.(*Store).TryUpdatePartitionMetadata(0xc0a5485c80, {0x8d5dbf8, 0xc0a5a94640}, {0xc0475f8ae0?, 0x560?, 0xc250ebe800?}, 0xac000?, {0x1, {0xc116f67000, 0x200, ...}, ...}, ...)
pkg/sql/vecindex/vecstore/store.go:379 +0x2e7
github.com/cockroachdb/cockroach/pkg/sql/vecindex/cspann.(*fixupWorker).updateMetadata(0xc12b3c1908, {0x8d5dbf8, 0xc0a5a94640}, 0x40d6f00fc7820001, {0x1, {0xc116f67000, 0x200, 0x200}, {0x9, 0xb6ca5efc7820001, ...}}, ...)
pkg/sql/vecindex/cspann/fixup_split.go:680 +0xe3
github.com/cockroachdb/cockroach/pkg/sql/vecindex/cspann.(*fixupWorker).splitPartition(0xc12b3c1908, {0x8d5dbf8, 0xc0a5a94640}, 0x1, 0x40d6f00fc7820001)
pkg/sql/vecindex/cspann/fixup_split.go:280 +0x1558
github.com/cockroachdb/cockroach/pkg/sql/vecindex/cspann.(*fixupWorker).Start(0xc12b3c1908, {0x8d5dbf8, 0xc0a5a94640})
pkg/sql/vecindex/cspann/fixup_worker.go:132 +0x2c5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x8d5dbf8, 0xc0a5a94640})
pkg/util/stop/stopper.go:447 +0x8a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 35156311
pkg/util/stop/stopper.go:445 +0x125
```

**Environment:**
- CockroachDB 26.2
- Server OS: Debian 13 (6.12.90+deb13.1-amd64)
- custom golang programming using `github.com/jackc/pgx/v5` (pg wire protocol)

**Additional context**
I also tried with batch sizes of 1000 and 500. While the "lock up" became less frequent, it didn't disappear. At lower batch sizes it was harder to reproduce.

If I omitted the vector index on the table I ran into no issues whatsoever.

I also noticed that sometimes I would see a failed transaction appear (when I looked at the insights page of the web panel) with the following information:

```
Error Code: 40001
Error Message: failed to update (remaining attempts=30): locking metadata for insert into partition ‹3337970724215324673›: getting partition metadata for ‹3337970724215324673›: TransactionRetryWithProtoRefreshError: WriteTooOldError: write for key /Table/552/6/‹1›/‹3337970724215324673›/‹0›/‹0› at timestamp 1782481469.623001427,0 too old; must write at or above 1782481469.704755794,1: "sql txn" meta={id=d884be18 key=/Table/552/6/‹1›/‹4637057731205922817›/‹0›/‹0› iso=Serializable pri=0.06669778 epo=0 ts=1782481469.704755794,1 min=1782481469.623001427,0 seq=0} lock=true stat=PENDING rts=1782481469.623001427,0 gul=1782481470.123001427,0 obs={n1@1782481469.623001427,0 n2@1782481469.684369613,0 n3@1782481469.692411452,0}
```

Jira issue: CRDB-65168

Start by reproducing the single-threaded batched insert with the vector index, then inspect the stack-trace paths in pkg/sql/vecindex/cspann/fixup_worker.go, pkg/sql/vecindex/cspann/fixup_split.go, and pkg/sql/vecindex/vecstore/store.go. Done means the workload no longer enters prolonged near-idle stalls or repeatedly fails with the reported 40001 error.

Written by the indexing model from the issue text.