apache / apache/iceberg-rust

_last_updated_sequence_number: physical per-row values discarded when first_row_id is null

Open
#2,991 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

None

### Describe the bug

### Context

When reading `_last_updated_sequence_number`, the Arrow reader gates on the data
file's `first_row_id`: if it is null, the whole column is nulled and any per-row
values the file physically carries are discarded (the physical leaf is never
projected).

Introduced in #2985 alongside the coalesce path.

### Behavior

This matches Java (`ValueReaders.lastUpdated` returns nulls when the base row id
is null), so it is not an interop regression. But the spec does not forbid such a
file, so it is a divergence between what the bytes can express and what we return.
Flagged in review as worth tracking rather than fixing in the coalesce PR.

### Options

- Keep the Java-parity behavior (current) ie a file with no row lineage has no
meaningful per-row sequence numbers.
- Pass non-null per-row values through, leaving null rows null when there is no
fallback. Note that this diverges from Java.

### To Reproduce

_No response_

### Expected behavior

_No response_

### Willingness to contribute

I can contribute a fix for this bug independently

Contributor guide

Open the contributing guide

Research direction

Start with the Arrow reader logic for _last_updated_sequence_number and the coalesce path introduced in #2985, focusing on how a null first_row_id affects physical per-row values. Resolve whether the implementation should preserve those values or retain Java-parity nulling, then add coverage demonstrating the chosen behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.