apache / apache/arrow-rs

[EPIC] [Parquet] Variant improvements and optimizations (not gating stabilization)

Open
#10,547 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Companion to https://github.com/apache/arrow-rs/issues/10546, which tracks the work gating the `variant_experimental` flag switch. This epic collects Variant work that is worth doing but does not block stabilization: performance, ergonomics, and internal consolidation.

**Optimizations**
- [ ] https://github.com/apache/arrow-rs/pull/10358
- [ ] https://github.com/apache/arrow-rs/pull/9862 (cast perfectly shredded children directly, closed for now, to be revived once the cast alignment under https://github.com/apache/arrow-rs/issues/8982 lands)
- [ ] https://github.com/apache/arrow-rs/issues/8153 (achievable today by chaining `variant_get` + `shred_variant`, direct reshredding is an optimization)
- [ ] https://github.com/apache/arrow-rs/issues/8323
- [ ] https://github.com/apache/arrow-rs/issues/7960

**Ergonomics and internals**
- [ ] https://github.com/apache/arrow-rs/issues/9519 (achievable today with multiple `variant_get` calls + manual assembly)
- [ ] https://github.com/apache/arrow-rs/issues/9633
- [ ] https://github.com/apache/arrow-rs/issues/8605
- [ ] https://github.com/apache/arrow-rs/issues/8987

Contributor guide

Open the contributing guide

Research direction

Start by reading the companion issue 10546 and the linked pull requests and issues listed under Optimizations and Ergonomics and internals. This is an epic rather than a single scoped change; done means selecting and completing one independently defined child item without blocking variant_experimental stabilization.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.