apache / apache/iceberg

Flink: Lookup join support

Open
#16,312 0 comments 1 reaction 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Feature Request / Improvement

I propose to resume work on the Lookup Join functionality for flink engine (https://github.com/apache/iceberg/pull/15183).
Ready to contribute based on closed PR and according the [last comment](https://github.com/apache/iceberg/pull/15183#pullrequestreview-3727232366)

### Motivation

Enable direct, efficient joins between high-throughput streams and slowly-changing dimension data stored in Iceberg tables.

### Implementation proposal

The Lookup Join implementation will be through the IcebergTableSourceFactory factory and the IcebergTableSource class, which create the IcebergLookupFunction search function. The main architecture includes IcebergLookupFunction for performing point queries on data and an optional IcebergLookupCache component for optimizing input-output.

Image

### Query engine

Flink

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start by reviewing closed PR #15183 and its last review comment, then inspect IcebergTableSourceFactory, IcebergTableSource, and the proposed IcebergLookupFunction architecture. Determine how the optional IcebergLookupCache fits the design. Done means Flink lookup joins can perform efficient point queries against Iceberg tables.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.