apache / apache/iceberg-rust

`arrow_schema_to_schema` fails to convert Arrow fields with UUID extension to Iceberg UUID

Open Beginner friendly
#2,929 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

None

### Describe the bug

Arrow supports a UUID extension type. When calling `arrow_schema_to_schema`, this extension is ignored. This adds friction when using UUID fields with Iceberg.

I propose that `arrow_schema_to_schema` takes the UUID extension into account, and when present, converts items to the Iceberg UUID type.

This adds friction because in practice we often need both Arrow and Iceberg schemas for fields. Which means either defining the fields twice by hand, or adding a wrapper around `arrow_schema_to_schema` to fixup the conversion (this is what I have had to do in my own work).

### To Reproduce

Here is a test for `crates/iceberg/src/arrow/schema.rs` which recreates the behaviour I would expect:

```rust
use arrow_schema::extension::Uuid as DataTypeUuidExt;

#[test]
fn test_arrow_schema_to_schema_should_convert_uuids() {
let converted_schema = arrow_schema_to_schema(&ArrowSchema::new(vec![
simple_field("uuid_field", DataType::FixedSizeBinary(16), false, "1")
.with_extension_type(DataTypeUuidExt),
]))
.unwrap();

let expected = Schema::builder()
.with_fields([NestedField::required(
1,
"uuid_field",
Type::Primitive(PrimitiveType::Uuid),
)
.into()])
.build()
.unwrap();

pretty_assertions::assert_eq!(expected, converted_schema);
}
```

### Expected behavior

* When I pass a field with the Arrow UUID type to `arrow_schema_to_schema`, the field is converted to the Iceberg `PrimitiveType::Uuid`.
* The Arrow UUID type is a `DataType::FixedSizeBinary(16)` with the Arrow UUID metadata of `ARROW:extension:metadata=arrow.uuid` set to it. This is based on the Arrow docs: https://arrow.apache.org/docs/format/CanonicalExtensions.html#uuid
* _Optional_ -- If a field has an Arrow `DataType` which is **not** `DataType::FixedSizeBinary(16)`, but does have the UUID metadata set, then it should raise an error.

### Willingness to contribute

I can contribute a fix for this bug independently

Contributor guide

Open the contributing guide

Research direction

Start in crates/iceberg/src/arrow/schema.rs and inspect arrow_schema_to_schema, then run or add the provided UUID conversion test using Arrow's UUID extension metadata. Done means a FixedSizeBinary(16) field marked with the UUID extension converts to PrimitiveType::Uuid; the optional invalid-type error behavior is also described for consideration.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.