Support different split criteria for consistent hashing index resizing
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently, we split at the middle of hash range when splitting hash buckets. This simple strategy may produce un-even children buckets when the data distribution is un-balanced.
A better strategy is to choose a split point which split bucket into a approximated even children buckets.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4377
- Type: New Feature
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with JIRA issue HUDI-4377 and locate Hudi's current consistent-hashing index bucket-splitting implementation. Read how the midpoint split is selected and find existing coverage for index resizing or bucket distribution. Done means the split criterion can produce approximately even child buckets for unbalanced data, with tests demonstrating the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100