[EPIC] [Parquet] Variant improvements and optimizations (not gating stabilization)
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 168
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Companion to https://github.com/apache/arrow-rs/issues/10546, which tracks the work gating the `variant_experimental` flag switch. This epic collects Variant work that is worth doing but does not block stabilization: performance, ergonomics, and internal consolidation.
**Optimizations**
- [ ] https://github.com/apache/arrow-rs/pull/10358
- [ ] https://github.com/apache/arrow-rs/pull/9862 (cast perfectly shredded children directly, closed for now, to be revived once the cast alignment under https://github.com/apache/arrow-rs/issues/8982 lands)
- [ ] https://github.com/apache/arrow-rs/issues/8153 (achievable today by chaining `variant_get` + `shred_variant`, direct reshredding is an optimization)
- [ ] https://github.com/apache/arrow-rs/issues/8323
- [ ] https://github.com/apache/arrow-rs/issues/7960
**Ergonomics and internals**
- [ ] https://github.com/apache/arrow-rs/issues/9519 (achievable today with multiple `variant_get` calls + manual assembly)
- [ ] https://github.com/apache/arrow-rs/issues/9633
- [ ] https://github.com/apache/arrow-rs/issues/8605
- [ ] https://github.com/apache/arrow-rs/issues/8987
Contributor guide
Research direction
Start by reading the companion issue 10546 and the linked pull requests and issues listed under Optimizations and Ergonomics and internals. This is an epic rather than a single scoped change; done means selecting and completing one independently defined child item without blocking variant_experimental stabilization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100