apache / apache/arrow-rs

Retrieve array from RecordBatch for a leaf column

Open
#5,699 9 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

While working filter pushdown for iceberg-rs: https://github.com/apache/iceberg-rust/pull/295, I am going to use the APIs like `ArrowPredicateFn` and `RowFilter`.

When constructing `ArrowPredicateFn` for iceberg predicate, we provide a filtering function that takes `RecordBatch` based on the given projection.

The `RecordBatch` contains the columns specified in the projection. And we need to access correct column in the batch to evaluate the predicate.

For top-level column, it should be straightforward. But for nested column, seems no way to access the particular array from the `RecordBatch`.

We only have the projection (i.e., `ProjectionMask`) which contains indices of leaf columns in the batch.

For example, if the schema has `[a, b, c]` top columns. `b` is a struct column with `[aa, bb, cc]` columns. Give a predicate like `cc > 1`, and we know the leaf indices of the nested column `cc` is 3.

Is there API we can use to access the array of `cc` in the `RecordBatch`?

**Describe the solution you'd like**

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by reading the RecordBatch, ProjectionMask, ArrowPredicateFn, and RowFilter APIs described in the issue, focusing on how leaf indices map to nested columns. Done means providing or documenting an API that retrieves the array for a nested leaf such as cc from the projected RecordBatch, with the behavior clear for top-level and nested columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.