Push down IS NULL / IS NOT NULL on struct columns into Parquet scan
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Filters like `WHERE struct_col IS NOT NULL` on struct columns cannot be pushed down into the Parquet scan. The `PushdownChecker` in `row_filter.rs` blanket-rejects any filter that references a whole struct column, even though `IS NOT NULL` only needs the struct's null bitmap — not any leaf data.
This forces a `FilterExec` above the scan that materializes ALL leaf columns of the struct just to check nullability:
```
FilterExec: price@4 IS NOT NULL
DataSourceExec: projection=[..., price, ...] -- reads all price.* leaf columns
```
For a struct with many fields, this means reading and decoding all leaf columns when semantically zero data is needed — only the struct's null bitmap.
PRs #20822 and #20854 added pushdown support for `get_field` expressions on struct fields (e.g., `s['value'] > 10`), but `IS NULL` / `IS NOT NULL` on the whole struct was explicitly left unsupported. The test `struct_data_structures_prevent_pushdown` asserts this rejection.
### Describe the solution you'd like
Allow `IS NULL` and `IS NOT NULL` on struct columns to be pushed down as row-level filters, reading only a single leaf column instead of all leaves.
In Parquet, definition levels encode nullability at every nesting level independently. When arrow-rs reads even one leaf column, it reconstructs the struct's null bitmap from definition levels. `is_not_null()` then checks this bitmap, not the leaf's data.
This means we can:
1. Detect `IS NULL(Column(struct))` / `IS NOT NULL(Column(struct))` in `PushdownChecker::f_down` before the `Column` node is visited
2. Project only the first leaf column of the struct in the `ProjectionMask`
3. Let arrow-rs reconstruct the struct null bitmap from that single leaf's definition levels
### Describe alternatives you've considered
1. **Definition-level-only reads**: Read only the definition levels of one leaf column without decoding the data values. This would be optimal (~1 bit per row vs 4+ bytes) but requires arrow-rs API changes (`ProjectionMask::definition_levels_only()` or similar) that don't exist today.
2. **Statistics-based pruning only**: Use Parquet row-group null_count statistics to prune entire row groups. However, Parquet statistics are stored only for leaf columns, not struct columns, so struct-level null_count isn't directly available from metadata.
### Additional context
Related to the filter pushdown EPIC: #20324
Builds on: #20822 (struct field pushdown) and #20854 (leaf projection refinement)
PR generated with [Claude Code](https://claude.com/product/claude-code)
Contributor guide
Research direction
Start in row_filter.rs at PushdownChecker::f_down and read the test struct_data_structures_prevent_pushdown, along with the related struct pushdown work in #20822 and #20854. Verify how the Parquet projection and arrow-rs reconstruction behave for a single struct leaf. Done means IS NULL and IS NOT NULL on whole struct columns push down while preserving correct results and avoiding projection of every leaf.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100