apache / apache/arrow-rs

[EPIC] A collection of items to improve speed of parquet metadata encoding

Open
#5,853 0 comments 0 reactions 0 assignees View on GitHub
enhancement parquet
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

There have been several recent assertions that Parquet is not suitable for handling wide tables with 1000s of columns

[The rationale](https://www.vldb.org/pvldb/vol17/p148-zeng.pdf) often goes something like wide tables have “large” metadata, which takes a “long time” to decode, often longer than reading the data itself.

This has led to several proposals for new file formats such as in [BtrBlocks](https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/papers/btrblocks.pdf), [Lance V2](https://blog.lancedb.com/lance-v2/). [Nimble,](https://github.com/facebookincubator/nimble) and recent [discussions](https://lists.apache.org/thread/8xmxc76nd00624qqps6s1qw6lhv1qwv5) on the parquet mailing list.

However, there are several ways we can improve the performance of the existing thrift decoding in parquet-rs and this ticket captures several ideas of how to do so

- [x] https://github.com/apache/arrow-rs/issues/5770
- [x] https://github.com/apache/arrow-rs/issues/5854
- [x] https://github.com/apache/arrow-rs/issues/5775
- [ ] https://github.com/apache/arrow-rs/issues/5855
- [x] https://github.com/apache/arrow-rs/issues/8518
- [x] https://github.com/apache/arrow-rs/issues/8537
- [x] https://github.com/apache/arrow-rs/issues/8572
- [ ] https://github.com/apache/arrow-rs/issues/8643
- [x] https://github.com/apache/arrow-rs/issues/5999
- [ ] https://github.com/apache/arrow-rs/issues/8713
- [ ] https://github.com/apache/arrow-rs/issues/7582

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the unchecked linked issues, especially 5855, 8643, 8713, and 7582, and compare them with the completed items in this epic. Determine which remaining metadata-encoding improvements are still actionable; the epic is done when its outstanding checklist items have been addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.