improve: reuse `Arc<dyn Array>` in parquet record batch reader.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
Both in `arrow_reader` and `async_reader`, if there are predicates, the reader will first read arrays (wrapped in `RecordBatch`) to evaluate the predicates and get a row selection. And then the reader will use the row selection to output the final needed arrays.
If some arrays in the final output are contained in the prefetched arrays, we also have to deserialize them again. This is quite wasteful.
We should have a reasonable way to reuse the arrays.
Contributor guide
Research direction
Start by reading the parquet record batch reader paths named `arrow_reader` and `async_reader`, focusing on predicate evaluation, prefetched `RecordBatch` arrays, and row selection. Define a reuse strategy that avoids deserializing arrays already available in the prefetched batches; done means both readers can reuse those arrays when they are part of the final output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100