apache / apache/fluss

[flink] Implement Fluss full-cache lookup join provider

Open
#3,726 0 comments 0 reactions 1 assignee Claimed by @loserwang1024 View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

Currently, Fluss Flink Connector supports lookup join through per-record lookup RPCs, optionally with Flink's partial lookup cache. In this mode, the connector does not preload dimension table data. Each cache miss triggers a lookup request to Fluss TabletServer.

This works for small traffic or low-cardinality lookup workloads, but it can become expensive in production when lookup QPS is high or the dimension table is relatively stable.

This issue aims to implement a Fluss full-cache lookup join provider, so lookup data can be loaded into local cache before serving lookup requests.

For many dimension table scenarios, users prefer full-cache lookup join:

```sql
SELECT s.order_id, s.user_id, d.user_name, d.level
FROM order_stream AS s
LEFT JOIN user_dim FOR SYSTEM_TIME AS OF s.proc_time AS d
ON s.user_id = d.user_id;
```

If user_dim is fully cached locally, lookup requests can be served without per-record RPC to Fluss, reducing latency and TabletServer pressure.

We have to support it by ourselves because flink still not support.
Image

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.