[Variant] Support extracting struct fields as Variant using ExtensionType
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
https://github.com/apache/arrow-rs/blob/5ba451531efd2e98de38f6a8443aad605b6b5cc5/parquet-variant-compute/src/variant_get.rs#L273-L276
Given data like:
```
{"user": {"id": 123, "metadata": {"tags": ["a"], "score": 95.5}}}
```
Users want to extract user with:
- id as Int32
- metadata as Variant (not expanded)
Currently, this requires multiple variant_get calls and manual StructArray assembly.
**Describe the solution you'd like**
Use ExtensionType to mark fields that should remain as Variant:
``` rust
let field = Field::new("user", DataType::Struct(Fields::from(vec![
Field::new("id", DataType::Int32, true),
Field::new("metadata", ..., true).with_extension_type(VariantType), // Keep as Variant
])), true);
let result = variant_get(&array, GetOptions::new().with_as_type(Some(Arc::new(field))))?;
```
Contributor guide
Research direction
Start at parquet-variant-compute/src/variant_get.rs around lines 273-276 and inspect how variant_get handles GetOptions::with_as_type. Verify how a nested Field with VariantType should preserve metadata as a Variant while extracting user.id as Int32; done means the requested StructArray result works for the example shape.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100