apache / apache/arrow-rs

Parquet Variant rejects canonical Arrow UInt8/UInt16/UInt32 typed values

Open
#10,416 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

### Describe the bug

The Arrow canonical extension specification for `arrow.parquet.variant` allows
`UInt8`, `UInt16`, and `UInt32` as Arrow `typed_value` storage types, with the
following logical Variant mappings:

| Arrow `typed_value` type | Variant primitive type |
|--------------------------|------------------------|
| `UInt8` | `Int16` |
| `UInt16` | `Int32` |
| `UInt32` | `Int64` |

However, `parquet-variant-compute` currently rejects these unsigned Arrow types
when constructing or shredding a `VariantArray`.

There is also an Arrow/Parquet schema boundary that needs special handling:
Parquet itself supports unsigned integer logical types, but the Parquet Variant
shredding specification does not allow unsigned `typed_value` columns. Therefore,
canonical unsigned Arrow storage must be lowered to the corresponding signed
Parquet shredded type when writing:

| Arrow `typed_value` | Parquet Variant shredded type |
|---------------------|--------------------------------|
| `UInt8` | `INT32` with `INTEGER(16, true)` |
| `UInt16` | plain `INT32` |
| `UInt32` | plain `INT64` |

### To Reproduce

```rust
use arrow_schema::DataType;
use parquet_variant::Variant;
use parquet_variant_compute::{shred_variant, VariantArray};

let input = VariantArray::from_iter([
Variant::Int16(0),
Variant::Int16(i16::from(u8::MAX)),
]);

let shredded = shred_variant(&input, &DataType::UInt8)?;
```

### Expected behavior

`VariantArray` and `shred_variant` should accept canonical Arrow `UInt8`, `UInt16`, and `UInt32` typed_value storage.

### Additional context

Arrow's documentation: https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings

Contributor guide

Open the contributing guide

Research direction

Start in parquet-variant-compute at VariantArray and shred_variant, using the UInt8 reproduction with arrow_schema::DataType as the entry point. Trace construction and shredding for UInt8, UInt16, and UInt32, then verify that Arrow values are accepted and lowered to the specified signed Parquet shredded types.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.