IDC-MO [Performance]: Pool destruction scans the global pointer registry and consumes substantial CN CPU

### Is there an existing issue for performance?

- [x] I have checked the existing issues.

### Environment

- IDC QA MO cluster, Kubernetes namespace `mo-new-moi`.
- Deployed image: `matrixone:v4.2.1-d2393868a-2026-08-28`.
- Commit: `d2393868a7aaa6343518d80849fe4696aa3577e9`.
- Two CNs: `moi-mo-2606-tp-cn-sktrz` and `moi-mo-2606-tp-cn-qs8zw`.
- Verified sktrz resources: CPU request 8 cores, limit 12 cores; memory request 20Gi, limit 55Gi.
- Evidence window: **2026-09-09 10:55–10:56 UTC / 18:55–18:56 UTC+8**.
- Workload: local Astra server using an isolated tenant on QA MO, with an immediate mock model response. Existing QA workloads remained running. This is not a full MOI end-to-end benchmark.

### Details of Performance

#### Confirmed expensive path: global pointer-registry scan during pool destruction

In the deployed revision, `MPool.destroy()` calls `OnHeapOutstanding()`. For a locking pool, this method walks **all process-wide pointer shards**, locks each shard, scans its map, and filters entries by pool ID. Pointer registration uses the same shard locks.

Consequently, destroying a pool performs work proportional to the process-wide allocation registry, not just the allocations owned by that pool. The implementation describes this as diagnostic-boundary work, but pool destruction occurs in background table-statistics tasks, connection cleanup, and operator cleanup.

Source at the deployed revision:

- [destroy and OnHeapOutstanding](https://github.com/matrixorigin/matrixone/blob/d2393868a7aaa6343518d80849fe4696aa3577e9/pkg/common/mpool/mpool.go#L588)
- [Global scan implementation](https://github.com/matrixorigin/matrixone/blob/d2393868a7aaa6343518d80849fe4696aa3577e9/pkg/common/mpool/mpool.go#L707)

This implementation was introduced on 4.2 by **#27498**, merged on August 23. GitHub ancestry comparison confirms that the deployed commit includes that change. This identifies the implementation's origin; it does **not** establish that all observed SQL latency was introduced by that PR.

#### Historical CPU-profile evidence

Existing Pyroscope profiles were retrieved for the exact one-minute window above, selecting `service_name="mo-new-moi/main"` and each CN pod, profile type `process_cpu:cpu:nanoseconds:cpu:nanoseconds`. Retrieval used `maxNodes=100000` to avoid substantial pruning into `other`.

| CN | Total sampled CPU seconds | OnHeapOutstanding self CPU | Selected scan subtree, including descendants |
|---|---:|---:|---|
| sktrz | 293.63 | 23.91 (8.1%) | Background statistics: **54.90 CPU seconds**, about **18.7%** of total sampled CPU |
| qs8zw | 386.90 | 13.62 (3.5%) | Connection cleanup: **24.12 CPU seconds** |

Main sktrz call path:

```text
dynamicCtx.betaTask.func2
-> dynamicCtx.statsCalculateOp
-> DeleteMPool
-> MPool.destroy
-> MPool.OnHeapOutstanding
```

Connection cleanup path on both CNs:

```text
MOServer.handleConn cleanup
-> Conn.Close
-> Routine.cleanup
-> Session.Close
-> DeleteMPool
-> MPool.destroy
-> MPool.OnHeapOutstanding
```

CPU seconds are summed across cores, **not request elapsed time**. Self cost and inclusive subtree cost overlap and must not be added.

#### User-visible impact and limits of the evidence

Four successive Astra requests with the immediate mock model all succeeded, but took **22.88s, 9.54s, 7.74s, 5.92s** end to end. The slow request included these MO statements:

| Statement ID | Operation | MO execution duration |
|---|---|---:|
| `01a085ce-d871-7f23-83f4-ba9e5ff256cf` | agent_runs event-counter UPDATE | 3190.62ms |
| `01a085ce-e073-779a-a8ee-57749c9b0f65` | agent_sessions UPDATE | 2929.37ms |
| `01a085ce-d120-75c4-8482-f1e6b09fbcbe` | session_artifacts UPDATE | 2395.73ms |
| `01a085ce-e563-7e25-963f-4bdc4ee9ab6c` | run_display_projections UPDATE | 2108.29ms |

These operations overlap; their durations must not be summed. Astra terminal-transaction connection acquisition plus BEGIN took only 19–28ms in these four requests. The session UPDATE's plan recorded substantial reader-building/file-service time and only 3.61ms of pre-run lock waiting.

**The global scan is a confirmed CPU overhead, but its exact contribution to each multi-second SQL operation has not been proven.** A later read-only goroutine snapshot did not catch an mpool lock waiter. SQL row-lock measurements do not measure the internal pointer-shard mutexes.

There is a separate concurrent storage-read latency signal: slow file-service logs on both CNs show selected `getReader` phases taking roughly 0.5–0.85s. These logs lack SQL trace IDs, so correlation is by CN/time window. This issue should not be interpreted as proving that removing the scan will eliminate all storage or SQL latency.

#### Expected behavior / suggested validation

- Pool destruction should not repeatedly scan the global allocation registry to determine one pool's outstanding ownership.
- Consider maintaining exact per-owner accounting, preserving allocation/free, cross-pool free, pool destruction, and ownership-diagnostic correctness; do not simply suppress diagnostics.
- Add a focused benchmark with many live allocations owned by unrelated pools while repeatedly creating/destroying a small pool. Measure CPU, teardown latency, and concurrent allocation/free latency as unrelated live allocations grow.
- Cover locking and noLock pools, on-heap/off-heap accounting, cross-pool frees, and outstanding allocations at teardown.
- Compare the same QA workload/profile after a fix, reporting tail latency separately from CPU savings.

### Additional information

The investigation was read-only: existing MO logs, Prometheus metrics, historical Pyroscope CPU profiles, deployed-revision source, and one current goroutine snapshot. No QA configuration or data was changed during this investigation. The profile/statement identifiers above allow maintainers to retrieve the original evidence from the IDC observability stack.