apache / apache/hudi

RLI & secondary index column file group counts can scale horizontally

Open
#16,912 1 comment 0 reactions 1 assignee Assigned to @yihua View on GitHub
from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

h3. Motivation
As of today for RLI we only set a fix number of file groups at all times. Individual file group size will keep expanding as the size of the record count increase. The size of individual file group will go unbounded which does not scale.
 
To address this issue, we need to figure out a way to dynamically repartition the existing file groups to a larger / smaller amount of file groups as needed. * Minimum downtime / exclusive access blocking all other operations while we scaling\
* basic correctness
* trigger of the scale up / down
* Integration with other operations on MDT

 
This should work for both RLI and secondary index on data columns.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9163
- Type: Bug
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.