tikv / tikv/pd

enhancement: add a bulk-load fast path for StorageEndpoint.LoadRegions

Open
#10,768 1 comment 0 reactions 0 assignees View on GitHub
contribution type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task

### Problem
Restoring PD with 100,015,214 regions took 18m14.087s in the process-only restart test. The hot path is `pkg/storage/endpoint/meta.go:174-221`, which does per-region `LoadRange -> proto.Unmarshal -> DecryptRegion -> core.NewRegionInfo -> CheckAndPutRegion` work, then walks overlap/subtree maintenance in `pkg/core/region.go:1095-1112` and `1257-1285`.

### Why this is slow
The restore path is effectively one region at a time. Even when the persisted data is already canonical and ordered, it still pays the full per-region parse / decrypt / tree update cost.

### Goal
Reduce 100M-scale restart recovery time without changing correctness.

### Possible directions
- Add a bulk-load fast path for ordered persisted regions.
- Reduce per-region allocations in `LoadRegions`.
- Avoid redundant overlap/subtree work when loading canonical data.
- Parallelize decode/decrypt before tree insertion if safe.

### Evidence
- 100M process-only restart: `load regions` completed in 18m14.087s.
- Code path: `pkg/storage/endpoint/meta.go:174-221`.
- Tree/subtree update path: `pkg/core/region.go:1095-1112`, `1257-1285`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.