apache / apache/arrow-rs

improve: reuse `Arc<dyn Array>` in parquet record batch reader.

Open
#4,864 7 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

Both in `arrow_reader` and `async_reader`, if there are predicates, the reader will first read arrays (wrapped in `RecordBatch`) to evaluate the predicates and get a row selection. And then the reader will use the row selection to output the final needed arrays.

If some arrays in the final output are contained in the prefetched arrays, we also have to deserialize them again. This is quite wasteful.

We should have a reasonable way to reuse the arrays.

Contributor guide

Open the contributing guide

Research direction

Start by reading the parquet record batch reader paths named `arrow_reader` and `async_reader`, focusing on predicate evaluation, prefetched `RecordBatch` arrays, and row selection. Define a reuse strategy that avoids deserializing arrays already available in the prefetched batches; done means both readers can reuse those arrays when they are part of the final output.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.