Reading v3 metadata with multi-argument transform fields (source-ids) fails
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Apache Iceberg Rust version
0.6.0 (latest version)
### Describe the bug
The V3 spec serializes partition fields and sort fields whose transform takes multiple arguments with a `source-ids` list instead of the single `source-id`. `PartitionField` and `SortField` only know `source-id`, so deserializing table metadata that contains any multi-argument transform field fails with `missing field source-id` — the whole v3 table becomes unreadable.
Per the spec, all v3 readers are required to read tables with unknown transforms, ignoring them.
For comparison, PyIceberg implemented this read tolerance in apache/iceberg-python#3630.
### To Reproduce
```rust
use iceberg::spec::PartitionField;
let field: Result = serde_json::from_str(
r#"{"source-ids": [1, 2], "field-id": 1000, "name": "m", "transform": "bucket[4]"}"#,
);
// Err: missing field `source-id` — expected the field to load with Transform::Unknown
```
Loading v3 table metadata containing such a partition field or sort field fails the same way.
### Expected behavior
Partition and sort fields with `source-ids` deserialize: a single-element list normalizes onto `source-id`; a multi-element list keeps the ids and treats the transform as `Transform::Unknown`. Serialization round-trips per the spec's writer rules (single-argument transforms write only `source-id`, multi-argument transforms write only `source-ids`).
Part of #2411 (Multi-arg Transforms → metadata read).
### Willingness to contribute
I can contribute a fix for this bug independently
Contributor guide
Research direction
Start with iceberg::spec::PartitionField and the corresponding SortField deserialization, using the provided serde_json reproduction as the first check. Compare the v3 metadata rules for source-id and source-ids, then verify that single-element and multi-element cases deserialize, unknown transforms are tolerated, and serialization round-trips according to the stated rules.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100