Support row group pruning for struct field predicates
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Related to the work on struct array handling:
- #20854
- #20822
- #20829
When filtering on struct fields (e.g. `WHERE s['value'] > 5`), Datafusion currently can not prune row groups using Parquet column statistics, even though the underlying leaf columns have valid min/max statistics stored in the parquet metadata
The issue is in the pruning predicate system. When it encounters a `GetField` expr like `GetField(Column("s"), "value")`, the column extraction logic only sees the parent struct `Column(s)` and doesn't resolve through to the nested field
Fixing this would mean teaching the pruning system to resolve `GetField` expressions down to their leaf columns, then look up the corresponding Parquet column stats. Note, the stats themselves are already there in the Parquet metadata, they're just never consulted for nested field access
On tables with many row groups, this could significantly reduce the amount of data read for struct field predicates
Contributor guide
Research direction
Start in the pruning predicate system, tracing how GetField(Column("s"), "value") is converted into column references and how Parquet column statistics are consulted. The work is done when struct field predicates resolve to their leaf columns and use the existing min/max statistics for row-group pruning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100