Cache parquet pruning setup across files with the same physical schema
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
In the parquet opener, DataFusion currently does per-file schema adaptation and pruning setup, including predicate rewrites and pruning predicate construction:
- https://github.com/apache/datafusion/blob/590a5178c8ffb17873f612a9c1da234fc1a18ff3/datafusion/datasource-parquet/src/opener.rs#L743-L788
- https://github.com/apache/datafusion/blob/590a5178c8ffb17873f612a9c1da234fc1a18ff3/datafusion/datasource-parquet/src/opener.rs#L1523-L1547
As @adriangb noted on https://github.com/apache/datafusion/pull/21480#issuecomment-4215673477, many deployments only have a small number of physical schemas, often just one, so repeating the same work across many files is wasteful.
PR #21480 from @fpetkovski improved this area by avoiding page pruning predicate construction unless page indexes are enabled, but we can do better and cache equivalent pruning setup across files with the same physical schema.
### Describe the solution you'd like
Cache parquet pruning setup across files when the physical schema and other correctness-relevant inputs are the same.
This likely includes:
- expression/schema rewrite results
- normal and pruning predicate construction
### Describe alternatives you've considered
Do nothing
### Additional context
Relevant links:
- Tracking comment from @adriangb:
https://github.com/apache/datafusion/pull/21480#issuecomment-4215673477
- Original PR from @fpetkovski:
https://github.com/apache/datafusion/pull/21480
- Page index loading / page pruning setup:
https://github.com/apache/datafusion/blob/590a5178c8ffb17873f612a9c1da234fc1a18ff3/datafusion/datasource-parquet/src/opener.rs#L793-L839
Contributor guide
Research direction
Start by reading the cited sections of datafusion/datasource-parquet/src/opener.rs, especially the per-file schema adaptation and pruning setup around lines 743-788 and 1523-1547, along with PR #21480. Determine which physical-schema and correctness-relevant inputs can share expression rewrites and predicate construction; done means equivalent files reuse that setup without changing normal or page-pruning behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100