apache / apache/datafusion

support dynamic filtering on partitioned data from file source

Open
#20,195 13 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

When preserve_file_partitions is enabled, we currently disable dynamic filtering to avoid incorrect assumptions about hash partitioning. This avoids a bug but removes critical filtering. We want to retain dynamic filtering without breaking file‑partitioning guarantees.

### Describe the solution you'd like

Enable dynamic filtering for file‑partitioned scans by pruning file groups based on partition values when the dynamic filter keys are a subset (which also implies exact match) of the partition columns.

This should allow dynamic filtering to work safely without requiring repartition or changing join planning.

### Describe alternatives you've considered

Introduce a new partitioning implementation via a generic trait‑based scheme (hash/value/range/custom), where users (and datafusion) can implement any type of partitioning scheme they desire.

This would provide the interface needed to determine how partitioning schemes are compatible, what satisfies what, etc. The exact details are not fleshed out but this would be a powerful addition and clear up ambiguities in DataFusion's partitioning modes today.

### Additional context

cc: @adriangb @NGA-TRAN @fmonjalet @gabotechs @LiaCastaneda

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by locating the preserve_file_partitions handling and dynamic-filter logic for file-partitioned scans; done means file groups are pruned when dynamic-filter keys are a subset of the partition columns without violating file-partitioning guarantees.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.