apache / apache/datafusion-ballista

Investigate use of `DynamicFilters` in ballista

Open
#1,375 7 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

There is great push in DataFusion community to support `DynamicFilters`, looks like there are [some good query improvements](https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/) when dynamic filters are used.

I'm not sure if they would improve ballista execution due to stage execution but at least we could investigate if they will yield any benefits for ballista

**Describe the solution you'd like**

It would be great if we could some effort to investigate if shuffle read would have any benefits of it.

- try to integrate dynamic filters with shuffle read to start with
- benchmark benefits (if any)
- provide road forward

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with the linked DataFusion DynamicFilters blog and trace Ballista's shuffle-read path. Try integrating dynamic filters there, benchmark whether they improve execution, and document the results and a recommended path forward.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.