Hot-region scheduling can see delayed CPU inflation after moving CPU-bound regions

## Summary

This issue records a hot-region scheduling phenomenon observed in our March 2026 CPU-dimension scheduling experiments: after moving hot peers/leaders away from a saturated TiKV store, the remaining source-side regions and the moved destination-side regions can both show much higher CPU than the CPU value observed at scheduling time.

This makes the CPU feedback for hot-region scheduling lag behind the real workload state. During that lag window, PD may continue to create more hot-region operators based on stale CPU observations, even though the destination store is about to become much hotter.

This issue only describes the observed phenomenon and the experiment where it was seen.

## Experiment

Experiment time: March 2026.

Experiment environment: internal TiKV hot CPU scheduling validation on testbed `8109116`.

Workload shape:

- Hot table: CPU-heavy, low-query/low-byte range scan workload.
- Hot SQL pattern: `SELECT SUM(SHA1(...) + ... x16) FROM sbtestN WHERE k BETWEEN start AND end`.
- Cold table: lightweight `COUNT(*)` range scan workload.
- The workload was intentionally designed to create hot regions dominated by unified-read-pool CPU, rather than by query count or bytes.
- Hot regions were initially concentrated on one TiKV store, then released to let hot-region scheduling move them.

The notes for this experiment were recorded around `2026-03-30` / `2026-03-31` under the hot CPU scheduling validation materials.

## Observed Phenomenon 1: remaining source-side regions became hotter after some hot regions were moved out

In one clean wave, PD dispatched several hot regions from store `1`:

| Time | Region | Movement | CPU observed at dispatch |
|---|---:|---|---:|
| `19:35:01` | `36532` | `1 -> 14` | `71` |
| `19:35:02` | `36536` | `1 -> 14` | `293` |
| `19:35:33` | `225232` | `1 -> 16` | `297` |

After these regions moved out, several regions still remaining on source store `1` became hotter:

| Region | CPU before / around dispatch | Later CPU on source |
|---:|---:|---:|
| `238484` | `244` at `19:35:00` | `348` at `19:35:43`, `586` at `19:37:13` |
| `225236` | `98` at `19:35:00` | `117` at `19:35:43`, `114` at `19:37:13` |
| `36544` | `90` at `19:35:00` | `111` at `19:37:13` |

So the source store did not immediately cool down in proportion to the moved-out CPU. Some remaining regions consumed more CPU after competing peers left the store.

## Observed Phenomenon 2: moved destination-side regions became much hotter after migration

The moved regions also did not keep the same CPU value after landing on a less saturated destination store.

Examples:

| Region | Dispatch | CPU observed at dispatch | Later observation on destination |
|---:|---|---:|---:|
| `36532` | `19:35:01`, `1 -> 14` | `71` | first appeared hot on store `14` at `19:38:45` with CPU `259` |
| `36536` | `19:35:02`, `1 -> 14` | `293` | first appeared on store `14` at `19:39:06` with CPU `82`, then rose to `697` at `19:39:55` |

For region `36532`, the delay from dispatch to obvious destination-side hot observation was about `224s`.

For region `36536`, the CPU later peaked far above the CPU observed at dispatch time.

## Observed Phenomenon 3: store-level CPU drop lagged behind operator creation by several minutes

Monitoring and logs both showed scheduling started around `19:35`, but the source store CPU did not drop clearly until around `19:39`.

The corresponding observation was that coprocessor wait duration remained high after the first scheduling wave, and CPU only dropped after wait duration decreased to a more reasonable level.

This means the scheduled peer movement was visible in PD/operator logs earlier than the real CPU feedback became visible in metrics.

## Why this is observable in CPU-bound workloads

The workload was CPU-bound. When a hot region runs on a saturated store, part of its demand is hidden by queueing and competition with other hot regions. After peer/leader movement changes the competition pattern:

- regions left on the source store can consume more CPU because fewer peers compete for the same unified read pool;
- regions moved to a less saturated destination store can consume more CPU than they did on the source store;
- the CPU measured at scheduling time can therefore under-estimate the CPU that the same region will consume after movement.

This is especially visible in the `sha1agg16` workload because request handling is mostly CPU computation. The same effect should be smaller for workloads whose bottleneck is not CPU.

## Impact Seen In The Experiment

During the observation lag, PD could continue to schedule more hot regions before the destination-side CPU had fully surfaced in metrics. This produced a risk of over-dispatching to stores that looked relatively idle at scheduling time but became hot shortly afterward.

The issue is not that an individual operator failed. The issue is that the observed CPU used for scheduling can be stale for CPU-bound hot regions immediately after movement, and the store-level feedback can arrive several minutes after the scheduling decision.