apache / apache/iceberg-rust

StaticTable::from_metadata_file() fails to read metadata for a table that was upgraded from v1 to v2

Open
#1,697 3 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

0.5.1

### Describe the bug

According to the [spec](https://iceberg.apache.org/spec/#version-2), the `sequence-number` field for a snapshot entry from v1 is optional/may not exist. When we try to read `metadata.json` that contains snapshots from both v1 and v2, the function fails with the following error if the v1 snapshot does not contain a `sequence-number`

```
DataInvalid => Failed to parse json string, source: data did not match any variant of untagged enum TableMetadataEnum
```

### To Reproduce

Using Iceberg Java SDK
- Create table, explicitly setting format version to 1
- Insert some rows into the table
- Change format version to 2, this does not result in full rewrite of files/manifests
- Add just one more row, creates file/manifest using version 2
- Try reading the metadata in Rust, and it fails with the error

### Expected behavior

Should we attempt to default the `sequence-number` to 0 (i.e. use `SnapshotV1`) to read such snapshot entries, if the `metadata.json` indicates that it belongs to v2?

### Willingness to contribute

I would be willing to contribute a fix for this bug with guidance from the Iceberg community

Contributor guide

Open the contributing guide

Research direction

Start at StaticTable::from_metadata_file() and inspect parsing of metadata.json snapshot entries, especially the optional sequence-number described in the issue. Reproduce a metadata file containing snapshots from both v1 and v2, then add coverage for the missing field. Done means upgraded v1-to-v2 metadata loads successfully without the reported TableMetadataEnum error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
63/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.