apache / apache/paimon

[Feature] Introduce key-value cache for paimon lookup operator in flink

Open
#3,428 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

When we use Paimon as the source for outer key joins, it is usually necessary to lookup the source table.

For example, there are two tables
1) Table `A(a, b, c, c1, c2, c3, c4, c5)`, where `a` is the primary key
2) Table `B(c, d, e, e1, e2, e3, e4, e5)`, where `c` is the primary key

Now we need to perform `A JOIN B` on `A.c = B.c` to output result `(a, b, c, d, e, c1, c2, c3, c4, c5, e1, e2, e3, e4, e5)`.

In Flink, we can convert the outer key join into a primary key join. We first perform `Join` on `A (a, c)` and `B (c)` to obtain the related data of `(a, c)`, and then lookup `A` and `B` respectively based on the `a` and `c` of the related data, and finally output the resulting data.
During this process, due to the delay (default 10 seconds) in loading incremental data of the Paimon dimension table, it is possible that the related data of `(a, c)` fails to lookup the data of `A` and `B` in a timely manner, resulting in incorrect output results.

To solve this issue, I'd like to introduce key-value cache in Paimon for lookup operator. When data is written to Paimon, it can be written to a key-value cache before the snapshot is created. And when the downstream operator get data from Paimon, it can always lookup data from key-value cache correctly.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the Paimon lookup-operator flow described in the issue and how incremental data reaches lookup consumers before a snapshot is created. Done means a key-value cache keeps newly written rows available to lookups without the stated delay and preserves correct outer-join results; the issue provides no files or tests to target.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.