apache / apache/datafusion

Cache parquet pruning setup across files with the same physical schema

Open
#21,554 1 comment 4 reactions 0 assignees View on GitHub
performance
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

In the parquet opener, DataFusion currently does per-file schema adaptation and pruning setup, including predicate rewrites and pruning predicate construction:
- https://github.com/apache/datafusion/blob/590a5178c8ffb17873f612a9c1da234fc1a18ff3/datafusion/datasource-parquet/src/opener.rs#L743-L788
- https://github.com/apache/datafusion/blob/590a5178c8ffb17873f612a9c1da234fc1a18ff3/datafusion/datasource-parquet/src/opener.rs#L1523-L1547

As @adriangb noted on https://github.com/apache/datafusion/pull/21480#issuecomment-4215673477, many deployments only have a small number of physical schemas, often just one, so repeating the same work across many files is wasteful.

PR #21480 from @fpetkovski improved this area by avoiding page pruning predicate construction unless page indexes are enabled, but we can do better and cache equivalent pruning setup across files with the same physical schema.

### Describe the solution you'd like

Cache parquet pruning setup across files when the physical schema and other correctness-relevant inputs are the same.

This likely includes:
- expression/schema rewrite results
- normal and pruning predicate construction

### Describe alternatives you've considered

Do nothing

### Additional context

Relevant links:
- Tracking comment from @adriangb:
https://github.com/apache/datafusion/pull/21480#issuecomment-4215673477
- Original PR from @fpetkovski:
https://github.com/apache/datafusion/pull/21480
- Page index loading / page pruning setup:
https://github.com/apache/datafusion/blob/590a5178c8ffb17873f612a9c1da234fc1a18ff3/datafusion/datasource-parquet/src/opener.rs#L793-L839

Contributor guide

Open the contributing guide

Research direction

Start by reading the cited sections of datafusion/datasource-parquet/src/opener.rs, especially the per-file schema adaptation and pruning setup around lines 743-788 and 1523-1547, along with PR #21480. Determine which physical-schema and correctness-relevant inputs can share expression rewrites and predicate construction; done means equivalent files reuse that setup without changing normal or page-pruning behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.