Introduce ProjectionMask To Allow Nested Projection Pushdown
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Currently projection indices are pushed down to scans as `Vec`. This creates some ambiguities:
* How to handle out of order or repeated indices - https://github.com/apache/arrow-datafusion/issues/2543
* How to handle nested types - https://github.com/apache/arrow-datafusion/issues/2453
To demonstrate how these problems intertwine, consider the case of
```
Struct {
first: Struct {
a: Integer,
b: Integer,
},
second: Struct {
c: Integer
}
}
```
If I project `["first.a", "second.c", "first.b"]` what is the resulting schema?
**Describe the solution you'd like**
I would like to propose we instead pushdown a leaf column mask, where leaf columns are fields with no children, as enumerated by a depth-first-scan of the schema tree. This avoids any ordering ambiguities, whilst also being relatively straightforward to implement and interpret.
I recently introduced a similar concept to the parquet reader https://github.com/apache/arrow-rs/pull/1716. We could theoretically lift this into arrow-rs, potentially adding support to RecordBatch for it, and then use this in DataFusion.
**Describe alternatives you've considered**
We could not support nested pushdown
**Additional context**
Currently pushdown for nested types in ParquetExec is broken - https://github.com/apache/arrow-datafusion/issues/2453
Thoughts @andygrove @alamb
Contributor guide
Research direction
Start by reading the ParquetExec nested projection-pushdown path and the referenced Arrow-RS parquet-reader change. Compare the current Vec behavior with the proposed leaf-column mask, then review issues 2453 and 2543 to understand the required schema and ordering semantics. Done should resolve the stated nested pushdown and ambiguity problems.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100