Field::try_canonical_extension_type has unexpected behavior from external writers
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
I created a sample Parquet file with pyarrow that contained Uuid columns.
Here is a test script:
```
use std::fs::File;
use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
fn main() {
let file = File::open("uuids.parquet").unwrap();
let reader = ParquetRecordBatchReaderBuilder::try_new(file).unwrap();
let schema = reader.schema();
let field = schema.fields.get(0).unwrap();
println!("{:?}", field);
match field.try_canonical_extension_type() {
Ok(extension_type) => println!("I am of extension type: {:?}", extension_type),
_ => println!("I am NOT an extension type")
}
}
```
and the output of `cargo run`:
```
$ cargo run
Field { name: "uuids", data_type: FixedSizeBinary(16), metadata: {"ARROW:extension:metadata": "", "ARROW:extension:name": "arrow.uuid"} }
I am NOT an extension type
```
This is because in the C++ definition of [UuidType](https://github.com/paleolimbot/arrow/blob/060062178ca85fa2d7dbd4083574bca6f91cc44c/cpp/src/arrow/extension/uuid.h#L52), the SerDe methods both return and expect the empty string. This causes issues in the Rust SerDe methods for [Uuid](https://github.com/apache/arrow-rs/blob/2108f20db1f6bc300bc6e1deacc0fca299e7feda/arrow-schema/src/extension/canonical/uuid.rs#L53), which return and expect Option::None.
I found this [comment](https://github.com/apache/arrow-rs/pull/5822/changes#r1926065958) in the original commit for the extension types. The simplest fix is just to accept the empty string as valid metadata.
Contributor guide
Research direction
Start with arrow-schema/src/extension/canonical/uuid.rs and the Field::try_canonical_extension_type entry point, then reproduce the metadata shown by the pyarrow-generated Parquet file. Confirm how empty extension metadata is handled and run the relevant arrow-schema tests. Done means externally written UUID fields are recognized as the canonical extension type without breaking existing cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100