apache / apache/arrow-rs

[Variant] Support extracting struct fields as Variant using ExtensionType

Open
#9,519 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

https://github.com/apache/arrow-rs/blob/5ba451531efd2e98de38f6a8443aad605b6b5cc5/parquet-variant-compute/src/variant_get.rs#L273-L276

Given data like:
```
{"user": {"id": 123, "metadata": {"tags": ["a"], "score": 95.5}}}
```

Users want to extract user with:

- id as Int32
- metadata as Variant (not expanded)

Currently, this requires multiple variant_get calls and manual StructArray assembly.

**Describe the solution you'd like**

Use ExtensionType to mark fields that should remain as Variant:
``` rust
let field = Field::new("user", DataType::Struct(Fields::from(vec![
Field::new("id", DataType::Int32, true),
Field::new("metadata", ..., true).with_extension_type(VariantType), // Keep as Variant
])), true);

let result = variant_get(&array, GetOptions::new().with_as_type(Some(Arc::new(field))))?;
```

Contributor guide

Open the contributing guide

Research direction

Start at parquet-variant-compute/src/variant_get.rs around lines 273-276 and inspect how variant_get handles GetOptions::with_as_type. Verify how a nested Field with VariantType should preserve metadata as a Variant while extracting user.id as Int32; done means the requested StructArray result works for the example shape.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.