apache / apache/paimon

[Feature] Strategically Engineering Wide Tables with Foreign Key-Based Joins

Open
#2,270 3 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

When performing a join operation with Flink's dual stream join, there is an issue with excessive state storage. Flink Lookup Join only concerns itself with changes in the primary stream; changes in the dimension table cannot update data that has already been joined. With Partial Update association, there is a problem where the joined tables need to have a common primary key to ensure matching between two or more data sources. All the aforementioned solutions for generating wide tables through multi-table associations have their limitations.

I would like to implement an entirely new method to overcome these shortcomings: a dynamic dimension table-driven Lookup Join, which I also refer to as 'ForeignKey Widening'.
https://cwiki.apache.org/confluence/display/PAIMON/PIP-12%3A+Strategically+Engineering+Wide+Tables+with+Foreign+Key-Based+Joins

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked PIP-12 design document, which describes the proposed ForeignKey Widening approach for dynamic dimension-table-driven joins. The issue names no source files, tests, entry points, or acceptance criteria, so the implementation scope and definition of done need to be established before work can begin.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.