Enhance the region balance of the 1M tables imported by lightning
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Development Task
### Background
- PD ensures that the table level's scatter, but it doesn't care about table-to-table, which relies on `balance-scheduler`.
- `Balance-Region` will not schedule empty region, There is a hardcode in
https://github.com/tikv/pd/blob/7e18a69734f2ae391516625ec0230c05b363f5c5/pkg/schedule/filter/region_filters.go#L150-L153
- When lightning imports tables, the table's region key is encoded in the table ID, while table IDs are created consecutively(basically next to each other). https://github.com/tikv/client-go/blob/6ba909c4ad2de65b5b36d0e5036d0a85f3154cc0/tikv/split_region.go#L241-L247
### Problems faced
For lightning importing 1 million tables(one table corresponds to one region), **even though** there are more than 3 stores, consecutive region keys will generate a lot of regions aggregations in the first 3 stores. And since regions are not scheduled, the three stores have a high probability of OOM.

Contributor guide
Assessment
This issue has not been assessed yet.