RLI & secondary index column file group counts can scale horizontally
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
h3. Motivation
As of today for RLI we only set a fix number of file groups at all times. Individual file group size will keep expanding as the size of the record count increase. The size of individual file group will go unbounded which does not scale.
To address this issue, we need to figure out a way to dynamically repartition the existing file groups to a larger / smaller amount of file groups as needed. * Minimum downtime / exclusive access blocking all other operations while we scaling\
* basic correctness
* trigger of the scale up / down
* Integration with other operations on MDT
This should work for both RLI and secondary index on data columns.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9163
- Type: Bug
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.