[Feature] Strategically Engineering Wide Tables with Foreign Key-Based Joins
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Motivation
When performing a join operation with Flink's dual stream join, there is an issue with excessive state storage. Flink Lookup Join only concerns itself with changes in the primary stream; changes in the dimension table cannot update data that has already been joined. With Partial Update association, there is a problem where the joined tables need to have a common primary key to ensure matching between two or more data sources. All the aforementioned solutions for generating wide tables through multi-table associations have their limitations.
I would like to implement an entirely new method to overcome these shortcomings: a dynamic dimension table-driven Lookup Join, which I also refer to as 'ForeignKey Widening'.
https://cwiki.apache.org/confluence/display/PAIMON/PIP-12%3A+Strategically+Engineering+Wide+Tables+with+Foreign+Key-Based+Joins
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked PIP-12 design document, which describes the proposed ForeignKey Widening approach for dynamic dimension-table-driven joins. The issue names no source files, tests, entry points, or acceptance criteria, so the implementation scope and definition of done need to be established before work can begin.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100