apache / apache/iceberg

Support merge-on-read tables in changelog scans

Open
#15,394 3 comments 1 reaction 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement
#10935 is working on adding delete file support to `IncrementalChangelogScan`, which will enable changelog scans for merge-on-read tables. This issue proposes a further optimization for v3 tables using row lineage.

### Motivation
With the delete file support from #10935, changelog scans will need to read and merge delete files with data files to identify changed rows. For v3 tables with row lineage, this can be done more efficiently by leveraging row-level metadata instead.

### Proposal
For v3 tables with row lineage, the changelog scan could use

- `_last_updated_sequence_number` to filter to only rows changed within the scan range, avoiding full data file reads and delete file merging
- `_row_id` to reliably match the same logical row across data files for UPDATE detection, without depending on identifier columns

Note that row lineage does not track changes made via equality deletes ([spec](https://iceberg.apache.org/spec/#row-lineage)), so the general delete file merging path from #10935 remains necessary as a fallback.

### Scope
This is an optimization on top of the delete file support in #10935, targeting v3 tables with row lineage only.

### Query engine
Spark / Core

### Willingness to contribute

- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start with IncrementalChangelogScan and the delete file work in #10935, then review the v3 row-lineage specification linked in the issue. Determine how _last_updated_sequence_number and _row_id can optimize changelog scans for row-lineage tables while retaining delete-file merging for equality deletes; done means the optimization is implemented with the fallback preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.