apache / apache/arrow-rs

Field::try_canonical_extension_type has unexpected behavior from external writers

Open Beginner friendly
#9,976 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

I created a sample Parquet file with pyarrow that contained Uuid columns.

Here is a test script:
```
use std::fs::File;
use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;

fn main() {
let file = File::open("uuids.parquet").unwrap();
let reader = ParquetRecordBatchReaderBuilder::try_new(file).unwrap();
let schema = reader.schema();
let field = schema.fields.get(0).unwrap();
println!("{:?}", field);
match field.try_canonical_extension_type() {
Ok(extension_type) => println!("I am of extension type: {:?}", extension_type),
_ => println!("I am NOT an extension type")
}
}
```

and the output of `cargo run`:
```
$ cargo run
Field { name: "uuids", data_type: FixedSizeBinary(16), metadata: {"ARROW:extension:metadata": "", "ARROW:extension:name": "arrow.uuid"} }
I am NOT an extension type
```

This is because in the C++ definition of [UuidType](https://github.com/paleolimbot/arrow/blob/060062178ca85fa2d7dbd4083574bca6f91cc44c/cpp/src/arrow/extension/uuid.h#L52), the SerDe methods both return and expect the empty string. This causes issues in the Rust SerDe methods for [Uuid](https://github.com/apache/arrow-rs/blob/2108f20db1f6bc300bc6e1deacc0fca299e7feda/arrow-schema/src/extension/canonical/uuid.rs#L53), which return and expect Option::None.

I found this [comment](https://github.com/apache/arrow-rs/pull/5822/changes#r1926065958) in the original commit for the extension types. The simplest fix is just to accept the empty string as valid metadata.

Contributor guide

Open the contributing guide

Research direction

Start with arrow-schema/src/extension/canonical/uuid.rs and the Field::try_canonical_extension_type entry point, then reproduce the metadata shown by the pyarrow-generated Parquet file. Confirm how empty extension metadata is handled and run the relevant arrow-schema tests. Done means externally written UUID fields are recognized as the canonical extension type without breaking existing cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.