StaticTable::from_metadata_file() fails to read metadata for a table that was upgraded from v1 to v2
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Apache Iceberg Rust version
0.5.1
### Describe the bug
According to the [spec](https://iceberg.apache.org/spec/#version-2), the `sequence-number` field for a snapshot entry from v1 is optional/may not exist. When we try to read `metadata.json` that contains snapshots from both v1 and v2, the function fails with the following error if the v1 snapshot does not contain a `sequence-number`
```
DataInvalid => Failed to parse json string, source: data did not match any variant of untagged enum TableMetadataEnum
```
### To Reproduce
Using Iceberg Java SDK
- Create table, explicitly setting format version to 1
- Insert some rows into the table
- Change format version to 2, this does not result in full rewrite of files/manifests
- Add just one more row, creates file/manifest using version 2
- Try reading the metadata in Rust, and it fails with the error
### Expected behavior
Should we attempt to default the `sequence-number` to 0 (i.e. use `SnapshotV1`) to read such snapshot entries, if the `metadata.json` indicates that it belongs to v2?
### Willingness to contribute
I would be willing to contribute a fix for this bug with guidance from the Iceberg community
Contributor guide
Research direction
Start at StaticTable::from_metadata_file() and inspect parsing of metadata.json snapshot entries, especially the optional sequence-number described in the issue. Reproduce a metadata file containing snapshots from both v1 and v2, then add coverage for the missing field. Done means upgraded v1-to-v2 metadata loads successfully without the reported TableMetadataEnum error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 63/100