apache / apache/fluss

Support lookup custom shuffle by Fluss bucket based on FLIP-462

Open
#3,727 0 comments 0 reactions 1 assignee Claimed by @loserwang1024 View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

Currently, Fluss lookup join does not support lookup custom shuffle. When lookup join is executed with normal hash shuffle, records are distributed by Flink's join-key hash, which is not aligned with Fluss bucket assignment.

This makes it difficult to combine lookup join with full-cache lookup efficiently. Each lookup subtask may receive keys from almost all Fluss buckets, so every subtask may need to load a large portion of the dimension table.

This issue aims to support lookup custom shuffle by Fluss bucket in Flink 2.2, so lookup input records can be shuffled according to Fluss bucket rules.

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.