apache / apache/datafusion-ballista
Investigate use of `DynamicFilters` in ballista
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
There is great push in DataFusion community to support `DynamicFilters`, looks like there are [some good query improvements](https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/) when dynamic filters are used.
I'm not sure if they would improve ballista execution due to stage execution but at least we could investigate if they will yield any benefits for ballista
**Describe the solution you'd like**
It would be great if we could some effort to investigate if shuffle read would have any benefits of it.
- try to integrate dynamic filters with shuffle read to start with
- benchmark benefits (if any)
- provide road forward
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start with the linked DataFusion DynamicFilters blog and trace Ballista's shuffle-read path. Try integrating dynamic filters there, benchmark whether they improve execution, and document the results and a recommended path forward.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100