apache / apache/datafusion

Support Runtime Pruning using Dynamic Filters and Inverted Index

Open
#19,764 2 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Using DataFusion 51 with HashJoin dynamic filters enabled, the performance gains are significant.
I’d like to extend this mechanism to leverage an inverted index, currently implemented in a custom TableProvider, for runtime pruning.

Architectural conflict:
Standard pruning occurs during logical/physical planning, but the values required for inverted index lookups (join keys - dynamic filter) are available only during the execution phase, after the build side of the hash join completes.

### Describe the solution you'd like

Introduce a custom execution node (e.g. IndexPruningExec) that wraps the DataSourceExec on the probe side of the join:
- Wait for dynamic filter values produced by HashJoinExec
- Query the inverted index using those values
- Dynamically prune files/row groups before executing the underlying DataSourceExec.

Questions
- Does this “wrapper exec” approach align with DataFusion’s long-term execution model?
- Is there an existing extension point intended for this kind of late-binding runtime pruning?
- Are there alternative designs (e.g., a pruning or predicate hook) that would be a better fit?

### Describe alternatives you've considered

Physical Optimizer Rules: Not viable because they run before dynamic filter values are produced.
Logical Sideways Information Passing: Too complex, duplicates work already done by the join's hash table, etc

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the execution flow around HashJoinExec and DataSourceExec, then examine how the custom TableProvider exposes its inverted index. Determine whether a wrapper such as the proposed IndexPruningExec or another extension point fits runtime dynamic filters, and document an agreed design before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.