Support reading _row_id and _last_updated_sequence_number metadata columns
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Is your feature request related to a problem or challenge?
We already write v3 row lineage (row ranges get assigned on the snapshot and
manifest during commit), but there's no way to read the two row-lineage
metadata columns back out during a scan:
- `_row_id`
- `_last_updated_sequence_number`
Both are defined in `metadata_columns.rs` but nothing actually produces their
values, so selecting `_row_id` in a scan currently errors out in the arrow read
pipeline, the field id passes projection validation and then there's no
constant/source for it.
For reference, Iceberg Java derives these on read rather than storing them per row.
`_last_updated_sequence_number` maps to the file's **data** sequence number:
the one that's preserved when a file is carried forward across a rewrite, not
the file sequence number (which changes on every commit that re-adds the file).
`_row_id` is `first_row_id +
position_in_file`, where the per-file `first_row_id` is inherited when the
manifest is read (manifest-level `first_row_id` plus a running record count).
We don't do any of that inheritance/derivation on the read side yet.
Note the write side already does the right thing here . v3 stores compact ranges
and expects readers to derive the per-row values, so this is really just a
missing piece on the read path.
This is a building block for row-lineage consumers (CDC/MERGE-style engines on
top of the scan API), not a changelog/CDC scan feature itself.
### Describe the solution you'd like
Happy to work on this. Part of the v3 row-lineage work (#2411) and complements
the write side (#1652); would follow the same metadata-column materialization
pattern as #2746 (`_pos`) and #2695 (`_spec_id`).
### Willingness to contribute
I can contribute to this feature independently
Contributor guide
Research direction
Start in metadata_columns.rs and trace the Arrow read pipeline and the metadata-column materialization patterns used for _pos (#2746) and _spec_id (#2695). Implement the read-side derivation and inheritance for _row_id and _last_updated_sequence_number, then verify that selecting both columns during a scan produces the defined per-row values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100