[EPIC] Improvements to struct access in Parquet
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
This EPIC tracks work on making queries over **struct columns in Parquet** read and evaluate only what they need — projection pruning, filter pushdown, row-group pruning, and the statistics that feed them.
Scope is the concrete Parquet-side machinery: `datafusion/datasource-parquet/src/{projection_read_plan,row_filter,nested_schema_pruning}.rs` and the physical expression adapter's interaction with it. Variant support builds on some of the same primitives but is tracked separately in #21308.
## Motivation
Two shapes reach the Parquet scan for struct access, and they are handled by unrelated code:
1. **`get_field` expressions** — `WHERE s['x'] = 200`, `SELECT s['a']`. Handled by `PushdownChecker` / `StructFieldAccess`.
2. **A narrowed declared schema** — the table declares `s STRUCT` where the file has `STRUCT`. The physical expression adapter rewrites the projected column into `CAST(s AS STRUCT)`, and engines like Spark/Comet communicate nested projection pruning this way, as a clipped read *schema* rather than as expressions.
Until #24090, shape (2) read every physical leaf and discarded the extras in memory. apache/datafusion-comet#4859 reported a production query reading **1.35 TB where plain Spark read 30.9 GB** for the same pruned `ReadSchema`. Any embedder that hands DataFusion a pre-pruned schema — Comet, delta-rs, Iceberg integrations — hits the same gap.
#24090 fixes that for the projection path. The items below are what it does not cover.
Note that the cast side of shape (2) is already settled: `CastColumnExpr` was removed in 54.0.0 (#21563) and struct-aware casting is unified into the field-aware `CastExpr`, which routes through `nested_struct::cast_column` via `ColumnarValue::cast_to`. That single cast node is what nested pruning clips against.
## Correctness
- [ ] #24109 — Parquet filter pushdown silently drops a `get_field` predicate when the file needs schema adaptation (**wrong results**). Blocks the filter-pushdown work below: on exactly the tables nested pruning targets, `WHERE s['field'] = ...` with `pushdown_filters = true` is wrong today.
## Projection & filter pruning
- [ ] #24090 — prune unread Parquet leaves when a nested column is cast to a narrower type (projection path)
- [ ] #24120 — extend nested schema pruning to filter pushdown
- [ ] #24121 — union the required leaves when a root is reached by multiple casts, or by a cast and a `get_field`, instead of falling back to a full read
- [ ] #24122 — clip `ListView` / `LargeListView` / `Dictionary` / `FixedSizeList` / `RunEndEncoded` wrappers, which are conservatively left unclipped today
- [ ] #21795 — push down `IS NULL` / `IS NOT NULL` on struct columns: `PushdownChecker` rejects whole-struct references, so the predicate materializes every leaf when only the definition levels are needed
- [ ] #23156 — refactor: build a reusable struct-access path tree for row-filter planning, so projection-mask selection and projected-schema pruning cannot drift
- [ ] #2581 — introduce `ProjectionMask` to allow nested projection pushdown (long-standing parent issue for this area)
## Row-group pruning & statistics
- [ ] #20871 — support row group pruning for struct field predicates
- [ ] #8334 — support Parquet statistics for struct columns
- [ ] #10609 — incorrect statistics read for struct array in Parquet (concrete reproducer for #8334; the bug that makes struct row-group pruning impossible today)
- [ ] #17221 — bloom filters and statistics not being used for Map keys/values
- [ ] #20707 — nested types and extension types are not supported in `Statistics` / `ColumnStatistics`
## Schema evolution
- [ ] #20835 — support additive schema evolution for `List` / nested container types in Parquet scans (in flight: #23914 for Map types)
- [ ] #15338 — `Unsupported cast with list of structs`: user-facing failure of the `nested_struct::cast_column` / `validate_struct_compatibility` path on `List` drift
- [ ] #20475 — read Parquet field ids for complex datatypes, so nested schema adaptation can match by field id rather than by name (Iceberg / Comet embedders)
## Shared with Variant
- #21306 — all struct-aware optimizations are hardcoded to `GetFieldFunc`; a custom UDF like `variant_get` gets none of them. Relevant to both this EPIC and #21308.
- #21307 — `ProjectionExec` produces unknown statistics for all `ScalarFunctionExpr` outputs, so every column reached through `get_field` / `variant_get` is invisible to the cost model.
## Related
- apache/datafusion-comet#4859 — the production report that motivated the projection-side work
- #21308 — first-class support for Variant access in Parquet
- #14755 — listing-table schema evolution depends on file order, which affects how struct schemas merge across files
- #2326 (closed) and #11745 (closed) — earlier attempts at this area, kept for history
Contributor guide
Research direction
Start by reading datafusion/datasource-parquet/src/{projection_read_plan,row_filter,nested_schema_pruning}.rs and the physical expression adapter, then use the linked child issues to choose a specific gap. Done means the selected struct-access path is covered without reading or evaluating unnecessary Parquet data, while preserving correctness for schema adaptation and filters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100