`arrow_schema_to_schema` fails to convert Arrow fields with UUID extension to Iceberg UUID
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Apache Iceberg Rust version
None
### Describe the bug
Arrow supports a UUID extension type. When calling `arrow_schema_to_schema`, this extension is ignored. This adds friction when using UUID fields with Iceberg.
I propose that `arrow_schema_to_schema` takes the UUID extension into account, and when present, converts items to the Iceberg UUID type.
This adds friction because in practice we often need both Arrow and Iceberg schemas for fields. Which means either defining the fields twice by hand, or adding a wrapper around `arrow_schema_to_schema` to fixup the conversion (this is what I have had to do in my own work).
### To Reproduce
Here is a test for `crates/iceberg/src/arrow/schema.rs` which recreates the behaviour I would expect:
```rust
use arrow_schema::extension::Uuid as DataTypeUuidExt;
#[test]
fn test_arrow_schema_to_schema_should_convert_uuids() {
let converted_schema = arrow_schema_to_schema(&ArrowSchema::new(vec![
simple_field("uuid_field", DataType::FixedSizeBinary(16), false, "1")
.with_extension_type(DataTypeUuidExt),
]))
.unwrap();
let expected = Schema::builder()
.with_fields([NestedField::required(
1,
"uuid_field",
Type::Primitive(PrimitiveType::Uuid),
)
.into()])
.build()
.unwrap();
pretty_assertions::assert_eq!(expected, converted_schema);
}
```
### Expected behavior
* When I pass a field with the Arrow UUID type to `arrow_schema_to_schema`, the field is converted to the Iceberg `PrimitiveType::Uuid`.
* The Arrow UUID type is a `DataType::FixedSizeBinary(16)` with the Arrow UUID metadata of `ARROW:extension:metadata=arrow.uuid` set to it. This is based on the Arrow docs: https://arrow.apache.org/docs/format/CanonicalExtensions.html#uuid
* _Optional_ -- If a field has an Arrow `DataType` which is **not** `DataType::FixedSizeBinary(16)`, but does have the UUID metadata set, then it should raise an error.
### Willingness to contribute
I can contribute a fix for this bug independently
Contributor guide
Research direction
Start in crates/iceberg/src/arrow/schema.rs and inspect arrow_schema_to_schema, then run or add the provided UUID conversion test using Arrow's UUID extension metadata. Done means a FixedSizeBinary(16) field marked with the UUID extension converts to PrimitiveType::Uuid; the optional invalid-type error behavior is also described for consideration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100