Low-Level Arrow Parquet Reader
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Currently we provide `ParquetRecordBatchReader` and `ParquetRecordBatchStream` as interfaces to read parquet data.
These interfaces are relatively straightforward to use, but have limitations:
* Cannot easily support concurrent decode (#5120)
* Limited flexibility w.r.t how data is fetched or decoded (#5141)
* Avoid decoding columns multiple times in the presence of predicates (#4864)
There is the experimental `ArrayReader` interface, however, this is very hard to use correctly and exposes a lot of what should probably remain implementation details.
**Describe the solution you'd like**
I would like an interface, perhaps similar in spirit to that added to the write side by #4871, that achieves the following:
* Makes it easy to parallelise both:
* The decoding of the parquet leaf columns
* The re-assembly of the arrow data from the dremel encodings
* Facilitates overriding the data source, e.g. by exposing the RowGroups trait
* Avoids exposing too many internal implementation details
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start by reading the existing ParquetRecordBatchReader, ParquetRecordBatchStream, and experimental ArrayReader interfaces, then review the related issues on concurrent decoding, data fetching, and predicate handling. Done means a new low-level reader interface supports parallel leaf decoding and Arrow re-assembly, allows the data source to be overridden through RowGroups, and hides unnecessary implementation details.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100