tikv / tikv/pd

Enhance the region balance of the 1M tables imported by lightning

Open
#8,424 3 comments 0 reactions 1 assignee Claimed by @JQWong7 View on GitHub
type/development
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Development Task

### Background

- PD ensures that the table level's scatter, but it doesn't care about table-to-table, which relies on `balance-scheduler`.
- `Balance-Region` will not schedule empty region, There is a hardcode in
https://github.com/tikv/pd/blob/7e18a69734f2ae391516625ec0230c05b363f5c5/pkg/schedule/filter/region_filters.go#L150-L153
- When lightning imports tables, the table's region key is encoded in the table ID, while table IDs are created consecutively(basically next to each other). https://github.com/tikv/client-go/blob/6ba909c4ad2de65b5b36d0e5036d0a85f3154cc0/tikv/split_region.go#L241-L247

### Problems faced

For lightning importing 1 million tables(one table corresponds to one region), **even though** there are more than 3 stores, consecutive region keys will generate a lot of regions aggregations in the first 3 stores. And since regions are not scheduled, the three stores have a high probability of OOM.

![img_v3_02cu_570929e1-9590-4330-8034-ed3a92434bfg](https://github.com/user-attachments/assets/adb2ceb9-259f-48f1-a33f-e30725fa29fc)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.