kv: chronic intent-cleanup wedge on hot-key UPSERT workload; scan-storm saturates leaseholder, lease migration and splits don't fix it

### Summary

On a long-running internal DRT chaos cluster, a high-concurrency UPSERT workload (gitload) produces a chronic, repeatable wedge on a single hot range (r138893) that recurs every ~20 minutes for >2 hours, even though:
- The cluster has no node failures, panics, OOMs, or disk stalls.
- Lease migrations (n3→n5→n4→n1) don't fix it — the wedge follows the lease.
- The splitter does fire (+25 ranges observed), but the post-split right-hand side still co-locates the workload's hot tables and inherits the contention.
- A goroutine dump during a brief recovery window shows the leaseholder is healthy at the goroutine layer (no lock-table waiters, no PushTxn pile-up, 17 runnable goroutines, healthy state distribution).
- Cleanup backlog at the SQL gateway node (78 `intentResolver.gcTxnRecord` goroutines all blocked in `requestbatcher.Send`) is consistent with a downstream slow range that the cleanup pipeline can't drain.

We don't have a confident root-cause hypothesis from the data we've gathered so far. Filing this as an investigation to:
1. Track the reproducer.
2. Ask for guidance on what additional profile (block / mutex / Pebble-level) would surface the actual bottleneck.

### Environment

- CockroachDB master, 2026-06-25
- 6-node AWS cluster, EBS storage
- Concurrent workloads on the same cluster:
- `tpcc` (warehouses=12000)
- `kv` (concurrency=128)
- `gitload` (UPSERT-heavy custom workload; 32 agents; `INSERT … ON CONFLICT DO UPDATE WHERE last_topo_order < excluded.last_topo_order` against ~500 hot PRIMARY KEY values on a newly-created `file_latest` table)
- Time window observed: 2026-06-25T13:14:00Z onwards (~2.5 h continuously)

### The reproducer

- 32 agents UPSERTing concurrently into a small PK hotspot (~500 keys on `file_latest`) + ~hundreds of related INSERTs into adjacent tables (`commits`, `commit_diffs`, `commit_parents`).
- All three tables landed on the existing trailing range r138893 when the gitload DB was first created, so r138893 spans `/{Table/4092/12/‹11763›/‹3›/‹2766›-Max}` covering both TPCC table 4092 and gitload tables 4102/4103/4106.
- Within ~10 min of workload start, r138893 begins exhibiting the wedge pattern.

### The wedge pattern (4 windows of observation)

Each ~20 min cycle:
1. `intentcount` climbs from ~100 to 2400–2700.
2. Hundreds of `slow range RPC` warnings fire on r138893: `ConditionalPut /Table/4106/1/"dir_N/file_M.txt"/0` taking 500–1193 s; `pusher … waiting … for pushee … PushTxn` calls timing out at 60–1190 s.
3. `closed_timestamp.max_behind_nanos` climbs monotonically (2.3s → 10.84s over 2 h, never recovers).
4. Lease eventually transfers (e.g., (n3,s9):8 → (n6,s24):6); the pile-up clears within seconds; cluster has a ~7–8 min "clean window."
5. Re-saturates within minutes; next cycle starts.

The pushee txn IDs rotate every cycle (5 distinct sets observed across 4 windows) but the contended **keys are stable** — gitload's hot UPSERT keys on `Table/4102` and `Table/4103`, plus a transient locker hot-spot on `Table/4106/1/"dir_2/file_2.txt"` in one window.

### Why "lock-table starvation" is NOT supported (corrected from earlier hypothesis)

Goroutine dumps from all three replicas of r138893 (n1 leaseholder, n4, n6) during a brief recovery window between wedge cycles show:
- **0 goroutines** in `lockTableWaiterImpl.WaitOn`
- **0 goroutines** in `PushTxn` paths
- **0 goroutines** in `concurrency.Manager.SequenceReq` or related lock-table code

State distribution on n1 is healthy: 1416 select / 200 sync.Cond.Wait / 102 IO wait / 17 runnable (out of 1837 total). No goroutine death spiral; no piled-up backend goroutines.

So the wedge is **not** visible at the goroutine layer on the leaseholder — at least not in the dump we captured.

### What IS visible at the goroutine layer (post-wedge backlog)

On node n4 (likely a heavy gateway node based on connection count), **78 distinct goroutines** are stuck in:

```
intentResolver.cleanupFinishedTxnIntents.func2
→ intentResolver.gcTxnRecord (intent_resolver.go:888)
→ requestbatcher.(*RequestBatcher).Send ← blocked here
[RunWithTimeout 20s]
```

Each is a post-commit txn-record GC request that can't land on its destination range. n1 (leaseholder) has 20 of these; n6 has 3. The asymmetry plus the consistent backlog pattern suggests these are accumulated backlog from a prior wedge cycle that's draining (consistent with our hypothesis that the dump was taken in a brief recovery window).

### Concrete observations from 4 successive 30-min Datadog windows

| Window | r138893 wedged | intentcount oscillating | Closed-ts lag | Lease moves | Splitter |
|---|---|---|---|---|---|
| 13:14–13:36 | yes (22 min) | n/a | 2.3s → 5.3s | (n3,s9):8 stuck, eventually moves to (n6,s24):6 | no split during window |
| 14:00–14:25 | yes | n/a | 5.3s → ~7s | replica churn every few min | no split |
| 14:34–15:04 | yes | 2 oscillation cycles, brief recovery 14:50–14:57 | ~7s → 9.71s | (n3,s9):8 → (n1,s1):19 | +25 ranges (split at ~14:54) |
| 14:53–15:23 | yes | 2 oscillation cycles | 9.06s → 10.84s | (n1,s1):19 (stable) → followers churn | brief +43 then -28 (split + merge) |

What's notable:
- Every window has the wedge active or recovering from it; not a single 30-min window has been clean.
- The split at 14:54 produced +25 ranges, but the post-split RHS still owns all gitload tables and re-wedged within minutes.
- The lease has migrated 3 times in 2 h. The wedge migrates with it.
- closed-ts lag is **monotonic and decoupled from intent state**: during the 8-min clean window at 15:06–15:16 when `intentcount` dropped to ~100, lag continued climbing at ~60ms/sec.

### Mechanism (high confidence; captured during an active wedge)

A subsequent active-wedge capture showed the bottleneck concretely:

**Range stats on the currently-hot gitload range:**
- `intent_count`: **1,116** active intents
- `key_count`: 360,111 total MVCC keys
- `live_count`: 86,283 live rows
- **key-to-live-row ratio: 4.2×** (each row has ~4 MVCC versions stacked on top)

**Leaseholder CPU profile (30s sample window during wedge):**
- ~7.7 cores actively burning in Pebble iteration:
- `pebble.(*Iterator).internalNextPrefix` cumulative **13.2%**
- `pebble.(*mergingIter).nextEntry` cumulative **10.6%**
- `cockroachkvs.Compare` cumulative 2.8%
- Not goroutine-blocked; actively scanning MVCC data.

**Stuck goroutines on leaseholder:**
- 256 goroutines in `intentResolver.cleanupFinishedTxnIntents` →
`intentResolver.gcTxnRecord` (intent_resolver.go:888) →
`requestbatcher.(*RequestBatcher).Send`, each holding a 20s `RunWithTimeout`.

**Mutex profile:**
- 59.9% of mutex delay cumulative through `kv.(*DB).sendUsingSender`
- 51.4% through `kv.(*Txn).Commit`
- Confirms the request batcher's lock serialization is the queue point.

**Reconstructed feedback loop:**
1. Concurrent UPSERTs on hot keys deposit write intents.
2. After commit, `cleanupFinishedTxnIntents` issues an MVCC scan to find each txn's intents to resolve.
3. Scans are expensive because hot keys carry many MVCC versions (4.2× ratio observed).
4. Scans saturate leaseholder CPU → cleanups don't drain → intent count climbs.
5. New UPSERTs scan past more intents → take longer.
6. Lease transfer doesn't help — wherever the lease goes, the same per-key scan cost reproduces.
7. Splits don't help — splitter fired naturally (+25 ranges observed in one 30-min window) but the post-split RHS inherited the hot keys.

The mechanism is per-key, not per-range. Manual pre-splitting was not tested.

### Asks

1. Is this interaction (per-txn `cleanupFinishedTxnIntents` issuing MVCC scans that saturate leaseholder CPU on a hot keyspace) a known issue or already tracked? Closest hit in our search was #165097 (parallel-commit + async intent resolution → spurious PushTxn timeouts), but the mechanism we observed is one layer deeper — the cleanup scans themselves, not the PushTxn waits.
2. Is there a cluster setting or admin hatch to:
- Throttle / batch / coalesce intent-cleanup scans on hot keyspaces
- Make `gcTxnRecord` async rather than blocking the cleanup goroutine for 20s
- Trigger aggressive MVCC GC on a specific hot range to drain accumulated versions
3. Is MVCC GC expected to keep the version-to-live ratio low under sustained hot-key UPSERT load? The 4.2× ratio we observed (key_count=360k vs live_count=86k) is what makes the scans expensive; if GC could keep this ratio at 1.5× or 2×, the scan cost would presumably stay manageable.
4. Manual pre-splitting was not tested. Would you expect aggressive pre-splits of the hot tables (e.g., ~50 splits across `file_latest`'s 500-key PK space) to mitigate, or does the per-key scan cost make sharding irrelevant?

### Related issues (for context — not asserting any of these match)

- #165097 (parallel-commit + async intent resolution → spurious PushTxn timeouts) — mechanism is plausible but we don't see goroutine-level evidence
- #99253 (PushTxn bypasses admission control → starves other work) — likely amplifier if the underlying mechanism is PushTxn-related
- #129579 (silently-slow leaseholder) — ruled out for this incident; leaseholder metrics are healthy
- #32367 (closed v19.1; same external symptom) — possible regression worth checking
- #51294 (closed; closed-ts climbing under contention) — related to symptom #3

### Reproduction notes

Observed continuously over 2.5 h on a long-running internal test cluster under the workload mix described above. We have not deliberately reproduced this in isolation. If a focused repro is useful, the rough recipe is:

- 6-node cluster with sparse range layout (recently-created tables on a single trailing range)
- TPCC + KV running steady-state
- Add an UPSERT-heavy workload doing 32-concurrent `INSERT … ON CONFLICT DO UPDATE WHERE …` against ~500 distinct PK values on a fresh table whose data initially lands in the existing trailing range
- Watch for `slow range RPC` warnings on that table's keyspace within ~10–20 min of workload start

Happy to provide a deterministic repro script if helpful — let me know.

Epic: none

Jira issue: CRDB-65153