kv,sql: expensive select queries cause OOM

**Describe the problem**

Running multiple queries in parallel can cause nodes to OOM.

**To Reproduce**

1) Create a 3 node cluster initialized with a KV DB.
2) Begin a background workload to write many KV keys
`roachprod ssh $CLUSTER:3 "./cockroach workload run kv $(roachprod pgurl $CLUSTER:3) --timeout 5s --tolerate-errors"`
3) Wait a little while to allow there to be enough keys written to the system.
4) Start a number of select queries in the background:
`for n in {1..3}; do for x in {1..100}; do echo "select * from kv.kv where v='abc';" | roachprod sql $CLUSTER:$n & done; done`
5) Notice that all the cockroach nodes crash
```
[17561.221476] cockroach invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[17629.744740] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-431.scope,task=cockroach,pid=32634,uid=1000
[17629.744773] Out of memory: Killed process 32634 (cockroach) total-vm:1735716kB, anon-rss:206688kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:964kB oom_score_adj:0
```

**Expected behavior**
The nodes should not crash. On a muti-tenant system a single tenant system, one tenant could bring down the entire cluster.

**Additional data / screenshots**
Depending on the number of concurrent select(*), we get different behaviors:
10 queries/node - p50 latency on writes jumps from 2ms -> 900ms, QPS goes from 4000 -> 10
20 queries/node - p50 latency goes to 2s, QPS goes to 1-2
40 queries/node - p50 latency goes to 10s, QPS goes to 0 (timeouts). Causes liveness failures.
100 queries/node - all nodes crash with OOM

CPU Profile: [profile.pb.gz](https://github.com/cockroachdb/cockroach/files/15099298/profile.pb.gz)
Heap profile (at 30 concurrency): [profile.pb.gz](https://github.com/cockroachdb/cockroach/files/15099391/profile.pb.gz)
Note that the heap profile doesn't account for all the memory. Here is a line from the cockroach-health log at about the same time as the heap profile:
`I240424 19:20:08.898205 324 2@server/status/runtime_log.go:47 ⋮ [T1,Vsystem,n2] 734 runtime stats: 6.2 GiB RSS, 982 goroutines (stacks: 17 MiB), 2.7 GiB/4.1 GiB Go alloc/total (heap fragmentation: 27 MiB, heap reserved: 1.3 GiB, heap released: 1.5 GiB), 2.1 GiB/2.4 GiB CGO alloc/total (9.0 CGO/sec), 394.5/3.1 %(u/s)time, 0.0 %gc (188x), 569 KiB/622 KiB (r/w)net`

**Environment:**
This likely cccurs on all releases. This was tested on 24.1/master.

**Additional context**
We have seen customer cases where heavy queries can cause either liveness failures or OOMs

Jira issue: CRDB-38160