apache / apache/arrow-rs

[Variant] Support dictionary-encoded value and typed_value inputs

Open
#11,068 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

### Is your feature request related to a problem or challenge?

Comet decodes dictionary-encoded Variant `value` and `typed_value` children before calling `VariantArray::try_new`. Such arrays can arise when a Parquet reader restores dictionary types from an embedded Arrow schema. #10802 / #10810 cover encoded **metadata** only.

This is a compatibility feature request: the [canonical Variant extension](https://github.com/apache/arrow/blob/main/docs/source/format/CanonicalExtensions.rst#parquet-variant) explicitly permits dictionary/run-end encoding for `metadata`, but does not currently permit these encodings for `value` or `typed_value`.

On arrow-rs `4cd8be954f6bc6b6dd265140207365b59a9900ec`, this reproducer rejects both inputs:

```rust
use std::sync::Arc;
use arrow::array::{Array, ArrayRef, BinaryArray, DictionaryArray, Int8Array, Int32Array, StructArray};
use arrow::datatypes::{Field, Int8Type};
use parquet_variant_compute::VariantArray;

fn main() {
let value: ArrayRef = Arc::new(DictionaryArray::::new(
Int8Array::from(vec![0]),
Arc::new(BinaryArray::from_vec(vec![&[12, 1]])),
));
let typed: ArrayRef = Arc::new(DictionaryArray::::new(
Int8Array::from(vec![0]), Arc::new(Int32Array::from(vec![1])),
));
for (name, encoded) in [("value", value), ("typed_value", typed)] {
let metadata: ArrayRef = Arc::new(BinaryArray::from_vec(vec![&[1, 0, 0]]));
let mut fields = vec![Arc::new(Field::new("metadata", metadata.data_type().clone(), false))];
let mut columns = vec![metadata];
if name == "typed_value" {
let value: ArrayRef = Arc::new(BinaryArray::from(vec![None::<&[u8]>]));
fields.push(Arc::new(Field::new("value", value.data_type().clone(), true)));
columns.push(value);
}
fields.push(Arc::new(Field::new(name, encoded.data_type().clone(), true)));
columns.push(encoded);
let input = StructArray::new(fields.into(), columns, None);
println!("{}", VariantArray::try_new(&input).unwrap_err());
}
}
```

Errors: `Illegal shredded value type: Dictionary(Int8, Binary)` and `Illegal shredded value type: Dictionary(Int8, Int32)`.

### Describe the solution you'd like

Support dictionary-encoded `value` and supported primitive `typed_value` inputs, including nested shredding states, through the shared Variant input path. Preserve logical nulls, parent nulls and sliced dictionary indices. If canonical storage remains strict, a supported normalization entry point would also let callers avoid maintaining their own recursive decoder.

### Describe alternatives you've considered

Comet currently decodes dictionaries recursively. For Parquet scans it also suppresses advisory Arrow schema hints to match Spark's physical interpretation; that can avoid these arrays at the reader boundary. Rejecting noncanonical encoded inputs is another valid policy, but encoded-metadata support alone does not justify removing the existing compatibility branch.

### Additional context

Tracked downstream in apache/datafusion-comet#5477; direct projection work is apache/datafusion-comet#5868. This concerns encoded inputs, distinct from dictionary output support in #10013 / #10014. Run-end-encoded `value`/`typed_value` is outside this request's immediate scope.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.