Parquet Variant rejects canonical Arrow UInt8/UInt16/UInt32 typed values
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
### Describe the bug
The Arrow canonical extension specification for `arrow.parquet.variant` allows
`UInt8`, `UInt16`, and `UInt32` as Arrow `typed_value` storage types, with the
following logical Variant mappings:
| Arrow `typed_value` type | Variant primitive type |
|--------------------------|------------------------|
| `UInt8` | `Int16` |
| `UInt16` | `Int32` |
| `UInt32` | `Int64` |
However, `parquet-variant-compute` currently rejects these unsigned Arrow types
when constructing or shredding a `VariantArray`.
There is also an Arrow/Parquet schema boundary that needs special handling:
Parquet itself supports unsigned integer logical types, but the Parquet Variant
shredding specification does not allow unsigned `typed_value` columns. Therefore,
canonical unsigned Arrow storage must be lowered to the corresponding signed
Parquet shredded type when writing:
| Arrow `typed_value` | Parquet Variant shredded type |
|---------------------|--------------------------------|
| `UInt8` | `INT32` with `INTEGER(16, true)` |
| `UInt16` | plain `INT32` |
| `UInt32` | plain `INT64` |
### To Reproduce
```rust
use arrow_schema::DataType;
use parquet_variant::Variant;
use parquet_variant_compute::{shred_variant, VariantArray};
let input = VariantArray::from_iter([
Variant::Int16(0),
Variant::Int16(i16::from(u8::MAX)),
]);
let shredded = shred_variant(&input, &DataType::UInt8)?;
```
### Expected behavior
`VariantArray` and `shred_variant` should accept canonical Arrow `UInt8`, `UInt16`, and `UInt32` typed_value storage.
### Additional context
Arrow's documentation: https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings
Contributor guide
Research direction
Start in parquet-variant-compute at VariantArray and shred_variant, using the UInt8 reproduction with arrow_schema::DataType as the entry point. Trace construction and shredding for UInt8, UInt16, and UInt32, then verify that Arrow values are accepted and lowered to the specified signed Parquet shredded types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100