apache / apache/arrow-rs

[Variant] unshred_variant panics when a shredded field name is absent from metadata

Open
#11,069 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

### Describe the bug

`unshred_variant` still panics when a present shredded object field has no entry in the row's metadata dictionary. `VariantArray::try_new` accepts the array, but the read-only metadata builder later returns an error that `ObjectBuilder::insert` unwraps.

Reproduced on arrow-rs `4cd8be954f6bc6b6dd265140207365b59a9900ec`, after #9741. This is malformed input under the current [Variant metadata requirement](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#variant-metadata), which requires shredded field names in metadata. The bug is the panic from a fallible kernel consuming file data.

### To Reproduce

```rust
use std::sync::Arc;
use arrow::array::{Array, ArrayRef, BinaryArray, Int32Array, StructArray};
use arrow::datatypes::Field;
use parquet_variant_compute::{VariantArray, unshred_variant};

fn structure(fields: Vec<(&str, ArrayRef, bool)>) -> StructArray {
let schema = fields.iter().map(|(name, array, nullable)|
Arc::new(Field::new(*name, array.data_type().clone(), *nullable)))
.collect::>();
StructArray::new(schema.into(), fields.into_iter().map(|(_, a, _)| a).collect(), None)
}

fn main() {
let missing: ArrayRef = Arc::new(BinaryArray::from(vec![None::<&[u8]>]));
let a: ArrayRef = Arc::new(structure(vec![
("value", missing.clone(), true),
("typed_value", Arc::new(Int32Array::from(vec![1])), true),
]));
let typed: ArrayRef = Arc::new(structure(vec![("a", a, false)]));
let input = structure(vec![
("metadata", Arc::new(BinaryArray::from_vec(vec![&[1, 0, 0]])), false),
("value", missing, true),
("typed_value", typed, true),
]);
let input = VariantArray::try_new(&input).unwrap();
let _ = unshred_variant(&input); // panics instead of returning Err
}
```

Panic: `called Result::unwrap() on an Err value: InvalidArgumentError("Field name 'a' not found in metadata dictionary")` at `parquet-variant/src/builder/object.rs:104`.

### Expected behavior

Return `ArrowError` for a present field whose name is absent from metadata, including nested objects and list elements. Absent object fields and rows masked by parent nulls should not trigger this error. Supporting permissive metadata repair would be a separate compatibility choice, not a requirement of this bug fix.

### Additional context

Related prior panic report: #9740 / #9741. Comet encounters this through Spark-compatible input handling and currently [extends metadata and remaps residual field IDs](https://github.com/apache/datafusion-comet/blob/6e556c944873554310aedd6a27f0d3e77ef17e71/native/core/src/parquet/cast_column/variant.rs#L586-L658) before unshredding in apache/datafusion-comet#5868.

Downstream tracking: apache/datafusion-comet#5477. Returning an error fixes the Arrow panic but does not by itself replace Comet's permissive Spark compatibility behavior.

Contributor guide

Open the contributing guide

Research direction

Start at parquet-variant/src/builder/object.rs:104 and trace how unshred_variant consumes the read-only metadata builder after VariantArray::try_new. Use the provided reproduction to confirm the panic and inspect handling of nested objects, list elements, absent fields, and parent-null rows. Done means present shredded fields missing from metadata return ArrowError without panicking, while valid absent or masked fields remain unaffected.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.