balance-hot-region and scatter-region are not fully aware of placement-rule scheduling domains
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Enhancement Task
### Background
Placement rules can partition TiKV stores into mutually exclusive store pools and assign different key ranges or replica roles to different pools.
This issue only describes scheduling behavior. It does not cover:
- Invalid or unsatisfiable placement rules.
- Scheduling storms caused by rule changes.
- Reduced recovery capacity within an isolated store pool.
### Summary
| Scheduler | Placement-rule-aware behavior | Non-placement-rule-aware behavior |
| --- | --- | --- |
| `balance-hot-region` | Placement safeguards filter illegal target stores | Expected load and standard deviation are calculated across all stores |
| `scatter-region` | Each peer replacement is checked by a placement safeguard | Multiple replacements are checked independently against the original Region fit |
### `balance-hot-region`
Hot-store statistics calculate one expected load and standard deviation across all stores and assign them to every store:
- [`store_hot_peers_infos.go`](https://github.com/tikv/pd/blob/c9fccfc6abe52aeaf538bf170f90af6181887c23/pkg/statistics/store_hot_peers_infos.go#L227-L292)
The hot-region solver uses this expectation when selecting source and destination stores:
- [Source-store filtering](https://github.com/tikv/pd/blob/c9fccfc6abe52aeaf538bf170f90af6181887c23/pkg/schedule/schedulers/hot_region_solver.go#L377-L416)
- [Destination-store filtering](https://github.com/tikv/pd/blob/c9fccfc6abe52aeaf538bf170f90af6181887c23/pkg/schedule/schedulers/hot_region_solver.go#L553-L662)
Placement safeguards are applied when selecting destinations, but they do not affect the expected load calculation.
For example, assume placement rules create two mutually exclusive store pools:
| Store pool | Store | Hot load |
| --- | --- | ---: |
| A | A1 | 120 |
| A | A2 | 80 |
| B | B1 | 0 |
| B | B2 | 0 |
The calculated expectations are:
```text
Cluster-wide expectation = (120 + 80 + 0 + 0) / 4 = 50
Pool A expectation = (120 + 80) / 2 = 100
```
With a tolerance ratio of `1.05`:
```text
A1 passes the source-store check:
120 > 50 * 1.05
A2 fails the destination-store check:
80 * 1.05 < 50 // false
```
B1 and B2 satisfy the load condition but are rejected by the placement safeguard because they are not eligible for Regions assigned to pool A.
In this case, `balance-hot-region` produces no operator for pool A.
It can work when the load baselines of different store pools are close enough for the cluster-wide expectation checks to pass. When the baselines differ significantly, a valid target inside a pool can be excluded by the global expectation.
### `scatter-region`
`RegionScatterer` calculates the original `RegionFit` once before selecting target peers:
- [`region_scatterer.go`](https://github.com/tikv/pd/blob/c9fccfc6abe52aeaf538bf170f90af6181887c23/pkg/schedule/scatter/region_scatterer.go#L620-L695)
Each peer replacement uses `NewPlacementSafeguard`, but the safeguard receives the original Region and the same original fit. Targets selected for previous replacements are not included in the fit used to check subsequent replacements.
The `selectedStores` map prevents selecting the same store ID more than once. It does not prevent selecting different stores with the same isolation label.
For example:
```text
Original peers:
Host A: A1
Host B: B1
Host C: C1
Candidate stores:
Host D: D1, D2, D3
```
The replacements can pass when checked independently against the original Region:
```text
A1 -> D1
B1 -> D2
C1 -> D3
```
The resulting peer set is:
```text
D1, D2, D3
```
All three stores belong to Host D, so the final peer set does not satisfy host-level isolation.
Scatter works when independently selected targets also happen to form a valid final placement. The individual placement checks do not guarantee that the complete target peer set is valid.
The same scatter behavior is tracked in:
- https://github.com/tikv/pd/issues/9953
Contributor guide
Research direction
Start with pkg/statistics/store_hot_peers_infos.go and the source- and destination-store filtering in pkg/schedule/schedulers/hot_region_solver.go, then inspect pkg/schedule/scatter/region_scatterer.go and the related issue #9953. Trace how placement-rule domains affect hot-load baselines and how successive scatter replacements are validated. Done means valid targets are not excluded by cross-pool expectations and the final scattered peer set satisfies placement isolation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100