apache / apache/hudi

add new index RANGE_BUCKET , when primary key is auto-increment like most mysql table

Open
#15,419 2 comments 0 reactions 1 assignee Claimed by @nsivabalan View on GitHub
from-jira priority:high status:pr-available type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

h3. Change Logs

Usually, in the mysql... table, there is an auto-increment primary key, base this fact and Bucket index , we propose a Range_Bucket index. And get a good performence in my practice.

Base concept is like Bucket index, and most important is: bucketId = primaryKey / stepSize
For example, if set stepSize = 2
bucketId Mapping will like this:
pKey, bucketId
1, 0
2, 0
3, 1
4, 1
5, 2
6, 2
7, 3
...

In my practice, I sync about 1000+ mysql table to hudi, and I set setpSize = 1,500,000, and basefile size will be about 50m - 350m.

Test like this, is set stepSize = 2 and pKey = \{1, 4, 9}, will get three base file:
[!https://user-images.githubusercontent.com/67826098/189126613-a50e3347-900f-4477-a4e8-f935bd996ebd.png|width=762!|https://user-images.githubusercontent.com/67826098/189126613-a50e3347-900f-4477-a4e8-f935bd996ebd.png]
h3. Impact

Introduce a new index RANGE_BUCKET, people can ust it like following:
 {{ option(HoodieIndexConfig.INDEX_TYPE.key, IndexType.BUCKET.name()).
option(HoodieIndexConfig.BUCKET_INDEX_ENGINE_TYPE.key, "RANGE_BUCKET").
option(HoodieIndexConfig.RANGE_BUCKET_STEP_SIZE.key, 2).
option(HoodieLayoutConfig.LAYOUT_TYPE.key, "BUCKET").
option(HoodieLayoutConfig.LAYOUT_PARTITIONER_CLASS_NAME.key, classOf[SparkRangeBucketIndexPartitioner[_]].getName).}}
*Risk level: none | low | medium | high*

low

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4824
- Type: New Feature

---

## Comments

09/Sep/22 17:19;wqwl611;https://github.com/apache/hudi/pull/6636;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.