_last_updated_sequence_number: physical per-row values discarded when first_row_id is null
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Apache Iceberg Rust version
None
### Describe the bug
### Context
When reading `_last_updated_sequence_number`, the Arrow reader gates on the data
file's `first_row_id`: if it is null, the whole column is nulled and any per-row
values the file physically carries are discarded (the physical leaf is never
projected).
Introduced in #2985 alongside the coalesce path.
### Behavior
This matches Java (`ValueReaders.lastUpdated` returns nulls when the base row id
is null), so it is not an interop regression. But the spec does not forbid such a
file, so it is a divergence between what the bytes can express and what we return.
Flagged in review as worth tracking rather than fixing in the coalesce PR.
### Options
- Keep the Java-parity behavior (current) ie a file with no row lineage has no
meaningful per-row sequence numbers.
- Pass non-null per-row values through, leaving null rows null when there is no
fallback. Note that this diverges from Java.
### To Reproduce
_No response_
### Expected behavior
_No response_
### Willingness to contribute
I can contribute a fix for this bug independently
Contributor guide
Research direction
Start with the Arrow reader logic for _last_updated_sequence_number and the coalesce path introduced in #2985, focusing on how a null first_row_id affects physical per-row values. Resolve whether the implementation should preserve those values or retain Java-parity nulling, then add coverage demonstrating the chosen behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100