apache / apache/hudi

Support different split criteria for consistent hashing index resizing

Open
#15,276 0 comments 0 reactions 0 assignees View on GitHub
area:index from-jira priority:high type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently, we split at the middle of hash range when splitting hash buckets. This simple strategy may produce un-even children buckets when the data distribution is un-balanced. 

 

A better strategy is to choose a split point which split bucket into a approximated even children buckets.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4377
- Type: New Feature

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with JIRA issue HUDI-4377 and locate Hudi's current consistent-hashing index bucket-splitting implementation. Read how the midpoint split is selected and find existing coverage for index resizing or bucket distribution. Done means the split criterion can produce approximately even child buckets for unbalanced data, with tests demonstrating the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.