apache / apache/datafusion

Support row group pruning for struct field predicates

Open
#20,871 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Related to the work on struct array handling:
- #20854
- #20822
- #20829

When filtering on struct fields (e.g. `WHERE s['value'] > 5`), Datafusion currently can not prune row groups using Parquet column statistics, even though the underlying leaf columns have valid min/max statistics stored in the parquet metadata

The issue is in the pruning predicate system. When it encounters a `GetField` expr like `GetField(Column("s"), "value")`, the column extraction logic only sees the parent struct `Column(s)` and doesn't resolve through to the nested field

Fixing this would mean teaching the pruning system to resolve `GetField` expressions down to their leaf columns, then look up the corresponding Parquet column stats. Note, the stats themselves are already there in the Parquet metadata, they're just never consulted for nested field access

On tables with many row groups, this could significantly reduce the amount of data read for struct field predicates

Contributor guide

Open the contributing guide

Research direction

Start in the pruning predicate system, tracing how GetField(Column("s"), "value") is converted into column references and how Parquet column statistics are consulted. The work is done when struct field predicates resolve to their leaf columns and use the existing min/max statistics for row-group pruning.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.