[Variant] Support list paths in ShreddedSchemaBuilder
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
### Is your feature request related to a problem or challenge?
`ShreddedSchemaBuilder` is syntactic sugar for constructing the `DataType` passed to `shred_variant`. It can construct nested structs, but valid paths containing an index currently reach `unreachable!`:
```rust
ShreddedSchemaBuilder::new()
.with_path("items[0].id", &DataType::Int64)?
```
Core list shredding is already supported; only schema construction through the builder is missing. This was deferred during #8940 until list shredding was implemented in #8831.
### Describe the solution you would like
Add a list node to `VariantSchemaNode` and allow paths to descend into the shared list element schema:
```text
items[0].id -> Struct>>
```
Possible path syntax:
- Use `[0]` as the conventional list-element marker, as suggested in #8940. Nonzero indices could be rejected or treated equivalently.
- Support JSONPath-style `[*]`, which expresses the intent more clearly. This would require either builder-specific parsing or a wildcard path element without changing `variant_get` index semantics.
Lists should be supported at any nesting level. Unsupported forms should return an error rather than panic.
### Additional context
- #8940
- #8831
- Current placeholder: https://github.com/apache/arrow-rs/blob/main/parquet-variant-compute/src/shred_variant.rs#L656-L658
Contributor guide
Assessment
This issue has not been assessed yet.