[Feature] Add support of pluggable Hash function for paimon bucket
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
Currently, paimon's bucket hash's hash value is based on `BinaryRow#hashcode`. It has two drawbacks.
1. The hash's logic is bind with the BinaryRow structure, So if we want to join the paimon table with another Hive table it's need a shuffle for the one side of the table, because the distribution hash function is different
2. To reduce the shuffle, if we want to use the paimon as bucket table, we need to reshuffle the hive table by the paimon's hash rule, but the BinaryRow based hash logic is hard to port to other engine.
So, I propose to make the hash function pluggable. In this way, we could introduce the HiveHash or the other to work with other compute engine.
### Solution
1. Introduce the hash function interface
```
public interface HashFunction {
int hash(BinaryRow row);
}
```
2. Adapt the read and writer to the new hash function interface
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing where BinaryRow#hashcode is used for Paimon bucket reads and writes. Review those paths against the proposed HashFunction interface and verify that both reading and writing use the selected hash implementation, including compatibility with a Hive-style hash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100