[Variant] unshred_variant panics when a shredded field name is absent from metadata
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
### Describe the bug
`unshred_variant` still panics when a present shredded object field has no entry in the row's metadata dictionary. `VariantArray::try_new` accepts the array, but the read-only metadata builder later returns an error that `ObjectBuilder::insert` unwraps.
Reproduced on arrow-rs `4cd8be954f6bc6b6dd265140207365b59a9900ec`, after #9741. This is malformed input under the current [Variant metadata requirement](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#variant-metadata), which requires shredded field names in metadata. The bug is the panic from a fallible kernel consuming file data.
### To Reproduce
```rust
use std::sync::Arc;
use arrow::array::{Array, ArrayRef, BinaryArray, Int32Array, StructArray};
use arrow::datatypes::Field;
use parquet_variant_compute::{VariantArray, unshred_variant};
fn structure(fields: Vec<(&str, ArrayRef, bool)>) -> StructArray {
let schema = fields.iter().map(|(name, array, nullable)|
Arc::new(Field::new(*name, array.data_type().clone(), *nullable)))
.collect::>();
StructArray::new(schema.into(), fields.into_iter().map(|(_, a, _)| a).collect(), None)
}
fn main() {
let missing: ArrayRef = Arc::new(BinaryArray::from(vec![None::<&[u8]>]));
let a: ArrayRef = Arc::new(structure(vec![
("value", missing.clone(), true),
("typed_value", Arc::new(Int32Array::from(vec![1])), true),
]));
let typed: ArrayRef = Arc::new(structure(vec![("a", a, false)]));
let input = structure(vec![
("metadata", Arc::new(BinaryArray::from_vec(vec![&[1, 0, 0]])), false),
("value", missing, true),
("typed_value", typed, true),
]);
let input = VariantArray::try_new(&input).unwrap();
let _ = unshred_variant(&input); // panics instead of returning Err
}
```
Panic: `called Result::unwrap() on an Err value: InvalidArgumentError("Field name 'a' not found in metadata dictionary")` at `parquet-variant/src/builder/object.rs:104`.
### Expected behavior
Return `ArrowError` for a present field whose name is absent from metadata, including nested objects and list elements. Absent object fields and rows masked by parent nulls should not trigger this error. Supporting permissive metadata repair would be a separate compatibility choice, not a requirement of this bug fix.
### Additional context
Related prior panic report: #9740 / #9741. Comet encounters this through Spark-compatible input handling and currently [extends metadata and remaps residual field IDs](https://github.com/apache/datafusion-comet/blob/6e556c944873554310aedd6a27f0d3e77ef17e71/native/core/src/parquet/cast_column/variant.rs#L586-L658) before unshredding in apache/datafusion-comet#5868.
Downstream tracking: apache/datafusion-comet#5477. Returning an error fixes the Arrow panic but does not by itself replace Comet's permissive Spark compatibility behavior.
Contributor guide
Research direction
Start at parquet-variant/src/builder/object.rs:104 and trace how unshred_variant consumes the read-only metadata builder after VariantArray::try_new. Use the provided reproduction to confirm the panic and inspect handling of nested objects, list elements, absent fields, and parent-null rows. Done means present shredded fields missing from metadata return ArrowError without panicking, while valid absent or masked fields remain unaffected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100