[EPIC] A collection of items to improve speed of parquet metadata encoding
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
There have been several recent assertions that Parquet is not suitable for handling wide tables with 1000s of columns
[The rationale](https://www.vldb.org/pvldb/vol17/p148-zeng.pdf) often goes something like wide tables have “large” metadata, which takes a “long time” to decode, often longer than reading the data itself.
This has led to several proposals for new file formats such as in [BtrBlocks](https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/papers/btrblocks.pdf), [Lance V2](https://blog.lancedb.com/lance-v2/). [Nimble,](https://github.com/facebookincubator/nimble) and recent [discussions](https://lists.apache.org/thread/8xmxc76nd00624qqps6s1qw6lhv1qwv5) on the parquet mailing list.
However, there are several ways we can improve the performance of the existing thrift decoding in parquet-rs and this ticket captures several ideas of how to do so
- [x] https://github.com/apache/arrow-rs/issues/5770
- [x] https://github.com/apache/arrow-rs/issues/5854
- [x] https://github.com/apache/arrow-rs/issues/5775
- [ ] https://github.com/apache/arrow-rs/issues/5855
- [x] https://github.com/apache/arrow-rs/issues/8518
- [x] https://github.com/apache/arrow-rs/issues/8537
- [x] https://github.com/apache/arrow-rs/issues/8572
- [ ] https://github.com/apache/arrow-rs/issues/8643
- [x] https://github.com/apache/arrow-rs/issues/5999
- [ ] https://github.com/apache/arrow-rs/issues/8713
- [ ] https://github.com/apache/arrow-rs/issues/7582
Contributor guide
Research direction
Start by reviewing the unchecked linked issues, especially 5855, 8643, 8713, and 7582, and compare them with the completed items in this epic. Determine which remaining metadata-encoding improvements are still actionable; the epic is done when its outstanding checklist items have been addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100