core: add an aggregated RegionStats index for arbitrary range queries
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Enhancement Task
### Problem
`RegionsInfo.GetRegionSizeByRange` still scans every Region in a non-empty range while holding the root Region tree read lock (`pkg/core/region.go:2231-2264`). Region heartbeat updates need the same `RegionsInfo.t` write lock in `setRegionLocked` (`pkg/core/region.go:1342-1396`), so the cost of a large range query grows with the number of Regions and can delay heartbeat processing.
The current mitigations are incomplete:
- [#9580](https://github.com/tikv/pd/pull/9580) makes the empty-to-empty full-range query O(1) by returning `regionTree.totalSize`.
- [#11072](https://github.com/tikv/pd/pull/11072) removes duplicate scans across Preparing stores and placement rules within one `checkStores` round.
- Non-empty ranges are still scanned in O(N) once per unique range and round.
- [#7252](https://github.com/tikv/pd/pull/7252) limits one lock hold to 1000 Regions, but does not change the total scan complexity.
This is a follow-up to [#9574](https://github.com/tikv/pd/issues/9574) and complements the broader Region tree contention discussion in [#9628](https://github.com/tikv/pd/issues/9628).
The same range-aggregation requirement is used by multi-tenant storage metering. `server/cluster/cluster.go:2737-2778` computes one range per enabled keyspace and calls `GetRegionStatsByRange`; the collector consumes `UserStorageSize` and `UserColumnarStorageSize`. Therefore this should be a reusable Region statistics index rather than a Preparing-only cache.
### Consistency requirement
Preparing progress and periodic storage metering consume approximate Region statistics. They do not require a point-in-time snapshot that is strongly consistent with the root Region tree. An asynchronously maintained index is acceptable if it:
- eventually converges to the root Region cache;
- does not permanently lose insert, delete, split, merge, or overlap replacement updates;
- repairs or rebuilds after a dropped/coalesced task or queue overload;
- exposes enough backlog/freshness information to diagnose excessive lag.
Temporary reordering of same-epoch statistic-only updates is acceptable. The index must not be used for a caller that requires exact point-in-time results without defining a stronger consistency contract for that caller.
### Goal
Provide a reusable, eventually consistent range-aggregation path with O(log N) query complexity for base Region statistics, independent of the number of Regions inside the requested range.
The first version should support:
- physical approximate Region size (`approximateSize`);
- user storage size (`approximateKvSize`);
- columnar user storage size (`approximateColumnarKvSize`);
- approximate keys, Region count, and empty-Region count.
The existing detailed statistics APIs that return Region/Peer/Store maps or hot statistics may continue to use a scan path.
### Existing asynchronous path that can be reused
The current subtree update path already provides most of the required execution model:
- the root Region tree is updated synchronously;
- `UpdateSubTree` is submitted to the existing heartbeat async runner;
- `CheckAndPutSubTree` re-reads the latest root Region by ID before applying an update;
- pending tasks with the same Region ID and task type are coalesced;
- insert and Region epoch changes are submitted as retained tasks;
- stale structural updates are rejected by term/version/confver checks.
The aggregate index should reuse this path, or use an equivalent path with the same semantics, rather than add mandatory index maintenance to the root heartbeat critical section.
The implementation still needs an explicit recovery rule. Once a task has left the pending queue, another task with the same Region ID can run concurrently, and non-retained tasks can be rejected under prolonged backlog. The existing subtree can trigger a later repair through its Region reference state; a separate scalar index must either participate in that repair path, maintain its own dirty state, or rebuild when it detects an unrecoverable gap.
### Candidate implementations
#### 1. Independent `regionStatsTree`
Maintain an augmented tree keyed by Region start key, protected independently from the root Region tree. Each node stores aggregate statistics for its descendants, while each Region entry stores only its range, epoch/term needed for stale-update rejection, and scalar statistics.
Updates should run through the asynchronous heartbeat subtree path and apply the latest absolute Region state rather than blindly applying heartbeat deltas. The path must cover:
- insert, delete, split, merge, and overlap replacement;
- same-range statistic-only heartbeat updates;
- `CheckAndPutRootTree` in the scheduling-service path;
- direct `SetRegion` and `RemoveRegion` calls;
- `ResetRegionCache` and Region reload.
This option isolates large range queries from the root Region lock and avoids synchronous aggregate maintenance on the heartbeat path, at the cost of a second key index and eventual consistency.
#### 2. Augment the existing root Region B-tree
Extend the existing B-tree nodes with subtree aggregate statistics. This avoids a second key index and can provide the same consistency as the root cache, but it requires maintaining aggregates across all B-tree mutation paths and adds update work while holding the root Region lock.
This option remains viable if benchmarks show that the additional heartbeat-path work and implementation complexity are acceptable.
### Query semantics
A range query should:
1. locate the first Region that contains or follows `startKey`;
2. locate the lower bound for `endKey`;
3. combine subtree aggregates instead of iterating each Region.
It must preserve the current `GetRegionSizeByRange` semantics: a Region intersecting the range contributes its full approximate size, including a Region containing `startKey` and a Region whose start key is before `endKey`.
Add a specialized API such as `GetRegionSizeStatsByRange` for callers that only need aggregate values. `getThreshold` should use the physical-size field, while storage metering should use the user and columnar fields. Detailed `/stats` and hot-region responses can retain their existing scan implementation.
### Evaluation and acceptance criteria
- Unit tests compare aggregate results with the existing scan implementation after asynchronous updates have drained, covering empty ranges, partial ranges, holes, split/merge, overlap replacement, statistic-only updates, zero sizes, and reset/reload.
- Tests inject delayed, reordered, coalesced, and rejected tasks and verify eventual repair or rebuild.
- Tests cover direct Region mutation paths and the scheduling-service heartbeat path.
- Concurrent heartbeat/update and range-query tests pass with the race detector.
- Benchmarks cover at least 100K, 1M, and multi-million Region populations, small/large/full ranges, multiple Preparing stores, and multiple keyspaces.
- Compare the independent async tree and augmented root B-tree on memory, heartbeat throughput/p99, root-lock wait, update cost, rebuild time, and query latency.
- Under concurrent heartbeat load, root lock wait and heartbeat p99 do not grow linearly with the queried range length.
- Measure convergence lag during normal load and overload; add pending/oldest-update or equivalent freshness metrics if existing runner metrics cannot establish it.
- Multi-tenant storage results converge to the current collector results, and collection time scales with the number of keyspaces rather than the number of Regions.
- Detailed Region/Peer/hot statistics remain behaviorally unchanged.
### Related issues and changes
- [#7248](https://github.com/tikv/pd/issues/7248) identified long Region-tree lock holds during Preparing progress calculation.
- [#9574](https://github.com/tikv/pd/issues/9574) tracks the current scale-out performance problem.
- [#9628](https://github.com/tikv/pd/issues/9628) tracks broader Region-tree query contention.
- [#9580](https://github.com/tikv/pd/pull/9580) optimizes the full-range case.
- [#11072](https://github.com/tikv/pd/pull/11072) removes repeated scans within one state-check round.
Contributor guide
Research direction
Start with pkg/core/region.go, especially GetRegionSizeByRange, setRegionLocked, UpdateSubTree, and CheckAndPutSubTree; then inspect server/cluster/cluster.go:2737-2778 and the existing heartbeat async runner. Compare the independent regionStatsTree and augmented root B-tree options before choosing an approach. Done means asynchronous aggregate queries converge with scan results across mutation and recovery paths, with race-tested unit coverage, benchmarks, and freshness metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100