apache / apache/datafusion

Introduce ProjectionMask To Allow Nested Projection Pushdown

Open
#2,581 18 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Currently projection indices are pushed down to scans as `Vec`. This creates some ambiguities:

* How to handle out of order or repeated indices - https://github.com/apache/arrow-datafusion/issues/2543
* How to handle nested types - https://github.com/apache/arrow-datafusion/issues/2453

To demonstrate how these problems intertwine, consider the case of

```
Struct {
first: Struct {
a: Integer,
b: Integer,
},
second: Struct {
c: Integer
}
}
```

If I project `["first.a", "second.c", "first.b"]` what is the resulting schema?

**Describe the solution you'd like**

I would like to propose we instead pushdown a leaf column mask, where leaf columns are fields with no children, as enumerated by a depth-first-scan of the schema tree. This avoids any ordering ambiguities, whilst also being relatively straightforward to implement and interpret.

I recently introduced a similar concept to the parquet reader https://github.com/apache/arrow-rs/pull/1716. We could theoretically lift this into arrow-rs, potentially adding support to RecordBatch for it, and then use this in DataFusion.

**Describe alternatives you've considered**

We could not support nested pushdown

**Additional context**

Currently pushdown for nested types in ParquetExec is broken - https://github.com/apache/arrow-datafusion/issues/2453

Thoughts @andygrove @alamb

Contributor guide

Open the contributing guide

Research direction

Start by reading the ParquetExec nested projection-pushdown path and the referenced Arrow-RS parquet-reader change. Compare the current Vec behavior with the proposed leaf-column mask, then review issues 2453 and 2543 to understand the required schema and ordering semantics. Done should resolve the stated nested pushdown and ambiguity problems.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.