[Parquet] Take ParquetRecordBatchReader in ReadPlanBuilder with_predicate
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
With the recent PRs to expose `ReadPlanBuilder` publicly, users can use `with_predicate` to evaluate the predicates. This is how it is being used today in arrow-rs:
```
let array_reader = ArrayReaderBuilder::new(&reader, &metrics)
.build_array_reader(fields.as_deref(), predicate.projection())?;
plan_builder = plan_builder.with_predicate(array_reader, predicate.as_mut())?;
```
Then a `ParquetRecordBatchReader::try_new` will be called to read parquet record batches created from array reader and the provided batch size.
However, in order to build an array reader publicly, there are several problems:
- ArrowReader metric enum is not publicly exposed in its mod
- `build_array_reader(fields.as_deref(), ...` I cannot easily get fields. I tried to get arrow field levels via `parquet_to_arrow_field_levels` but the internal fields from the levels are also private
**Describe the solution you'd like**
Instead of taking array reader in the `with_predicate` method and create a new ParquetRecordBatchReader inside, take `ParquetRecordBatchReader` as input parameter instead.
**Describe alternatives you've considered**
Expose all the required fields and methods as public to be able to create an array reader.
There is alternative to expose a method like below
```
pub fn with_predicate_and_level_row_groups(
mut self,
levels: &FieldLevels,
row_groups: &dyn RowGroups,
predicate: &mut dyn ArrowPredicate,
) -> Result {
let mut reader = ParquetRecordBatchReader::try_new_with_row_groups(levels, row_groups, self.batch_size, self.selection.clone(), predicate.projection().clone())?;
```
**Additional context**
Contributor guide
Research direction
Start at ReadPlanBuilder::with_predicate and trace how ParquetRecordBatchReader::try_new consumes the ArrayReader built by ArrayReaderBuilder::build_array_reader. Compare the requested reader-based API with the alternative row-group method, then verify that predicate evaluation still produces the expected record batches without requiring private fields or metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100