apache / apache/arrow-rs

[Variant] Support list paths in ShreddedSchemaBuilder

Open
#10,615 1 comment 0 reactions 1 assignee Claimed by @cakeni View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

### Is your feature request related to a problem or challenge?

`ShreddedSchemaBuilder` is syntactic sugar for constructing the `DataType` passed to `shred_variant`. It can construct nested structs, but valid paths containing an index currently reach `unreachable!`:

```rust
ShreddedSchemaBuilder::new()
.with_path("items[0].id", &DataType::Int64)?
```

Core list shredding is already supported; only schema construction through the builder is missing. This was deferred during #8940 until list shredding was implemented in #8831.

### Describe the solution you would like

Add a list node to `VariantSchemaNode` and allow paths to descend into the shared list element schema:

```text
items[0].id -> Struct>>
```

Possible path syntax:

- Use `[0]` as the conventional list-element marker, as suggested in #8940. Nonzero indices could be rejected or treated equivalently.
- Support JSONPath-style `[*]`, which expresses the intent more clearly. This would require either builder-specific parsing or a wildcard path element without changing `variant_get` index semantics.

Lists should be supported at any nesting level. Unsupported forms should return an error rather than panic.

### Additional context

- #8940
- #8831
- Current placeholder: https://github.com/apache/arrow-rs/blob/main/parquet-variant-compute/src/shred_variant.rs#L656-L658

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.