apache / apache/datafusion

Extend Parquet nested schema pruning to filter pushdown

Open
#24,120 2 comments 0 reactions 1 assignee Claimed by @blinding-pixels View on GitHub
datasource enhancement performance
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Part of #24119.

## Background

apache/datafusion#24090 teaches the Parquet **projection** path to prune leaves when a nested column is consumed through a narrowing cast — the shape `DefaultPhysicalExprAdapter` produces whenever a table's declared nested schema is narrower than the physical file (Comet, delta-rs, Iceberg integrations all hit this). `PushdownChecker` collects a `CastColumnAccess` for `CAST(col AS narrower_nested_type)`, and `clip_for_cast` walks the physical and target type trees together to compute exactly which Parquet leaves the cast consumes.

That collection is deliberately opt-in and enabled **only for projection analysis**:

```rust
/// Enable collection of whole-column casts to narrower nested types.
pub(crate) fn with_cast_collection(mut self) -> Self { ... }
```

Filter pushdown leaves it off, so a predicate over a narrowed nested column still reads every physical leaf of that column.

## What this issue asks for

Extend the same clipping to the row-filter path (`datafusion/datasource-parquet/src/row_filter.rs`), so that with `pushdown_filters = true` a predicate like `WHERE s['x'] = 200` against a table whose declared schema narrows `s` reads only the leaves the predicate needs, rather than the whole column.

The projection side already computes this; the filter side needs the equivalent two outputs kept in agreement:

1. the `ProjectionMask` handed to the reader for the filter, and
2. the projected Arrow schema the row filter evaluates against.

## Prerequisite: apache/datafusion#24109

This cannot land as a pure optimization until #24109 is resolved. Today `PushdownChecker::f_down` only recognises `get_field` whose first argument is a bare `Column`. When the expression adapter interposes a cast — which is *always* the case on exactly the tables this feature targets — the visitor descends past the `CastExpr`, sees a bare struct `Column`, sets `non_primitive_columns = true`, and `pushdown_columns` returns `None`. The conjunct is then silently dropped from the row filter while planning has already removed `FilterExec` on the strength of an "exactly handled" claim made against the *table* schema.

So filter pushdown over a narrowed nested column is currently **unsound, not merely unoptimised**. Teaching `PushdownChecker` to look through the cast would address both #24109 and this issue, but only if the post-decode fallback proposed in #24109 exists to catch whatever the row-filter construction still cannot handle.

## Suggested order

1. Fix #24109 (post-decode filter fallback, so a failed row-filter build costs performance rather than correctness).
2. Enable cast collection in the filter path and route clipped leaves through row-filter planning.
3. Reconcile with the struct-access path handling in `row_filter.rs` — see #23156, which proposes a shared access tree for exactly the "mask and projected schema must describe the same pruned shape" invariant.

## Notes

- Related: apache/datafusion-comet#4859, the production report that motivated the projection-side work (1.35 TB read vs 30.9 GB for the same pruned `ReadSchema`).
- The clip is total by design: any shape it does not understand keeps all leaves, so the worst case is today's full read. That property should be preserved on the filter side.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.