apache / apache/paimon

[Feature] Add support of pluggable Hash function for paimon bucket

Open
#5,436 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

Currently, paimon's bucket hash's hash value is based on `BinaryRow#hashcode`. It has two drawbacks.

1. The hash's logic is bind with the BinaryRow structure, So if we want to join the paimon table with another Hive table it's need a shuffle for the one side of the table, because the distribution hash function is different
2. To reduce the shuffle, if we want to use the paimon as bucket table, we need to reshuffle the hive table by the paimon's hash rule, but the BinaryRow based hash logic is hard to port to other engine.

So, I propose to make the hash function pluggable. In this way, we could introduce the HiveHash or the other to work with other compute engine.

### Solution

1. Introduce the hash function interface

```
public interface HashFunction {

int hash(BinaryRow row);

}
```
2. Adapt the read and writer to the new hash function interface

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing where BinaryRow#hashcode is used for Paimon bucket reads and writes. Review those paths against the proposed HashFunction interface and verify that both reading and writing use the selected hash implementation, including compatibility with a Hive-style hash.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.