apache / apache/fluss

[lake][iceberg] Iceberg does not support union read for primary key table

Open
#3,048 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

0.9.0 (latest release)

### Please describe the bug 🐞

I find Iceberg does not support union read primary key table, IcebergLakeSource#createRecordReader returns IcebergRecordReader which not a subclass of SortedRecordReader, so LakeSnapshotAndLogSplitScanner will fail when pollBatch.

I check `testReadIcebergLakeTable` in `FlinkUnionReadPrimaryKeyTableITCase`, seems it just read records from $lake, but not the union read.

Is there something I'm misunderstanding?

### Solution

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with IcebergLakeSource#createRecordReader and compare IcebergRecordReader with the expectations of LakeSnapshotAndLogSplitScanner during pollBatch. Inspect testReadIcebergLakeTable in FlinkUnionReadPrimaryKeyTableITCase and verify whether it exercises union reads; done means union reading a primary-key Iceberg table succeeds without the scanner failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.