apache / apache/arrow-rs

Low-Level Arrow Parquet Reader

Open
#5,522 3 comments 8 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Currently we provide `ParquetRecordBatchReader` and `ParquetRecordBatchStream` as interfaces to read parquet data.

These interfaces are relatively straightforward to use, but have limitations:

* Cannot easily support concurrent decode (#5120)
* Limited flexibility w.r.t how data is fetched or decoded (#5141)
* Avoid decoding columns multiple times in the presence of predicates (#4864)

There is the experimental `ArrayReader` interface, however, this is very hard to use correctly and exposes a lot of what should probably remain implementation details.

**Describe the solution you'd like**

I would like an interface, perhaps similar in spirit to that added to the write side by #4871, that achieves the following:

* Makes it easy to parallelise both:
* The decoding of the parquet leaf columns
* The re-assembly of the arrow data from the dremel encodings
* Facilitates overriding the data source, e.g. by exposing the RowGroups trait
* Avoids exposing too many internal implementation details

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing ParquetRecordBatchReader, ParquetRecordBatchStream, and experimental ArrayReader interfaces, then review the related issues on concurrent decoding, data fetching, and predicate handling. Done means a new low-level reader interface supports parallel leaf decoding and Arrow re-assembly, allows the data source to be overridden through RowGroups, and hides unnecessary implementation details.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.