apache / apache/iceberg-rust

Support reading _row_id and _last_updated_sequence_number metadata columns

Open
#2,879 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

We already write v3 row lineage (row ranges get assigned on the snapshot and
manifest during commit), but there's no way to read the two row-lineage
metadata columns back out during a scan:

- `_row_id`
- `_last_updated_sequence_number`

Both are defined in `metadata_columns.rs` but nothing actually produces their
values, so selecting `_row_id` in a scan currently errors out in the arrow read
pipeline, the field id passes projection validation and then there's no
constant/source for it.

For reference, Iceberg Java derives these on read rather than storing them per row.
`_last_updated_sequence_number` maps to the file's **data** sequence number:
the one that's preserved when a file is carried forward across a rewrite, not
the file sequence number (which changes on every commit that re-adds the file).
`_row_id` is `first_row_id +
position_in_file`, where the per-file `first_row_id` is inherited when the
manifest is read (manifest-level `first_row_id` plus a running record count).
We don't do any of that inheritance/derivation on the read side yet.

Note the write side already does the right thing here . v3 stores compact ranges
and expects readers to derive the per-row values, so this is really just a
missing piece on the read path.

This is a building block for row-lineage consumers (CDC/MERGE-style engines on
top of the scan API), not a changelog/CDC scan feature itself.

### Describe the solution you'd like

Happy to work on this. Part of the v3 row-lineage work (#2411) and complements
the write side (#1652); would follow the same metadata-column materialization
pattern as #2746 (`_pos`) and #2695 (`_spec_id`).

### Willingness to contribute

I can contribute to this feature independently

Contributor guide

Open the contributing guide

Research direction

Start in metadata_columns.rs and trace the Arrow read pipeline and the metadata-column materialization patterns used for _pos (#2746) and _spec_id (#2695). Implement the read-side derivation and inheritance for _row_id and _last_updated_sequence_number, then verify that selecting both columns during a scan produces the defined per-row values.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.