resource control: RU throttling can cause TiDB OOM by retaining large gRPC response buffers

## Bug Report

### 1. Minimal reproduce step (Required)

We observed this in production and do not yet have a fully automated minimal reproducer. The sanitized workload shape is:

1. Assign sessions to a low-RU, non-burstable resource group.
2. Repeatedly run an `IndexHashJoin` query whose leading side is filtered by a large `IN (...)` list and whose inner-side KV responses can be multiple MiB.
3. Submit about one statement per second while earlier statements are still incomplete, allowing tens of logical executions and hundreds of IndexHashJoin inner workers/RPCs to overlap.
4. Capture TiDB heap and goroutine profiles while RU throttling is active.

Representative query shape:

```sql
SELECT /*+ LEADING(t1, t2) INL_HASH_JOIN(t2) */
t1.key, t2.value
FROM t1
JOIN t2 ON t1.join_key = t2.join_key
WHERE t1.key IN (...);
```

The production incident had Select QPS below 1, but the last complete goroutine profile still contained 85 concurrent/incomplete execution identifiers for the dominant digest.

### 2. What did you expect to see? (Required)

RU throttling should bound forward progress without allowing untracked TiDB-side KV/gRPC response buffers to exhaust process memory.

Specifically:

- received/materialized response bytes should be visible to the statement and global memory trackers;
- TiDB should apply backpressure to new coprocessor/Index Join work when the resource group has a large response-wait backlog;
- process heap should remain bounded even when many statements are slowed by RU waits.

### 3. What did you see instead (Required)

TiDB heap grew from 10.38 GiB to 21.75 GiB in 12 minutes and the process restarted/OOMed. The last complete pre-OOM heap profile was dominated by gRPC response materialization:

| Time | Heap in-use | `BufferSlice.Materialize` | `NopBufferPool.Get` |
| --- | ---: | ---: | ---: |
| T+0 min | 10.38 GiB | 4.55 GiB | 2.53 GiB |
| T+6 min | 19.68 GiB | 10.72 GiB | 4.53 GiB |
| T+12 min | 21.75 GiB | 13.22 GiB | 3.09 GiB |
| After restart | 0.10 GiB | baseline | baseline |

At T+12 min:

```text
13.22 GiB 60.79% google.golang.org/grpc/mem.BufferSlice.Materialize
3.09 GiB 14.23% google.golang.org/grpc/mem.NopBufferPool.Get
1.55 GiB 7.14% IndexNestedLoopHashJoin.constructLookupContent (cumulative)
1.03 GiB 4.75% buildKvRangesForIndexJoin (cumulative)
```

The matching goroutine profile contained 8,931 goroutines:

- 5,000 goroutines were tagged with one SQL digest;
- 85 distinct execution identifiers were present for that digest;
- 1,525 target-digest goroutines were in `interceptedClient.SendRequest`;
- 583 were IndexHashJoin inner workers;
- 157 instance-wide goroutines were in `OnRequestWait`;
- 177 instance-wide goroutines were in `OnResponseWait`;
- 83 target-digest goroutines were in `OnResponseWait`;
- 318 instance-wide goroutines were in `sendToRespCh`.

This is a TiDB process-memory failure. We did not observe a TiKV or coprocessor process OOM.

### 4. What is your TiDB version? (Required)

- TiDB Cloud build with a v8.5.4 compatibility/version label
- profiled TiDB build ID: `f4e552cce7537cfdf7c6e644eae72624d2de6c14`
- client-go revision: `5526ae2a85466d085df6fe89d63331aa21e0a4d7`
- PD client revision: `491a11c2f061b3dac6e442eb33d9c645ebe05889`
- grpc-go: `v1.75.1`

## Profile- and source-backed mechanism

In the profiled client-go revision, `interceptedClient.SendRequest` receives the response first, then calls `OnResponseWait`, and only returns the response after the RU wait. The response therefore remains reachable across the wait:

https://github.com/tikv/client-go/blob/5526ae2a85466d085df6fe89d63331aa21e0a4d7/internal/client/client_interceptor.go#L68-L87

The PD resource-group controller acquires response-side RU tokens in `OnResponseWait`; responses of at least 4 MiB are treated as large:

- https://github.com/tikv/pd/blob/491a11c2f061b3dac6e442eb33d9c645ebe05889/client/resource_group/controller/group_controller.go#L856-L875
- https://github.com/tikv/pd/blob/491a11c2f061b3dac6e442eb33d9c645ebe05889/client/resource_group/controller/global_controller.go#L45-L53

grpc-go `BufferSlice.Materialize` allocates a contiguous byte slice and copies the response buffers. The compatibility codec calls this path while unmarshalling:

- https://github.com/grpc/grpc-go/blob/v1.75.1/mem/buffer_slice.go#L88-L96
- https://github.com/grpc/grpc-go/blob/v1.75.1/codec.go#L62-L79

The resulting sequence is:

```text
IndexHashJoin fan-out / large lookup work
-> many KV responses
-> gRPC materializes response bytes in TiDB
-> RU request/response waits slow completion
-> response buffers and join workers remain live across more executions
-> TiDB heap reaches the process limit
-> OOM/restart
```

The immediate heap cause is directly confirmed by pprof. RU throttling as a lifetime/concurrency amplifier is supported by matching goroutine profiles and the source ordering above. Heap samples do not carry a SQL digest for every allocated byte, so this report does not claim byte-perfect attribution of all 13.22 GiB to one digest.

## Suggested fix areas

1. Account KV/gRPC response buffers and materialized decode buffers to the statement and global TiDB memory trackers.
2. Bound new coprocessor/Index Join work based on response-waiting count or bytes for the resource group.
3. Avoid holding fully materialized responses during `OnResponseWait`, or charge/admit before materialization where protocol semantics allow.
4. Export metrics for response-waiting bytes/count by resource group and, where practical, SQL digest.
5. Add a regression test with repeated IndexHashJoin statements, large responses, and low non-burstable RU; assert bounded heap and tracked memory.

## Related issues / changes

- #68545: oversized Index Join chunks and unexpected memory usage
- #69049 and #69965: Index Join chunk-size fix and release-8.5 backport
- #68959: v8.5.4 benchmark showing significant heap/RSS reduction from Index Join memory changes
- #67817: untracked range-building memory for long `IN` predicates
- tikv/pd#8460: response-side RU wait for large requests/responses

These appear related but not duplicate: the distinct failure here is TiDB OOM from a large population of materialized gRPC/KV responses whose lifetime is extended under RU throttling.

Raw profiles contain internal cluster and SQL labels and can be shared privately with maintainers.

Start by reading client_interceptor.go around SendRequest and the PD resource-group controller paths linked in the report, then inspect the cited pprof profiles and related Index Join issues. A useful validation path is the proposed repeated IndexHashJoin workload with large responses and low non-burstable RU. Done means response memory is tracked or bounded, response waits do not retain unbounded buffers, and a regression test demonstrates this.

Written by the indexing model from the issue text.