[Epic] Dynamic filtering related items
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
This is a collection of various items related to "dynamic filtering"
Roughly speaking dynamic filters are filters who values are known during runtime but not planning (for example from the values of [topk](https://github.com/apache/datafusion/issues/15037))
Here is some more background:
* [Snowflake blog: Reimagining Top-K Aggregation Optimization at Snowflake](https://www.snowflake.com/en/engineering-blog/optimizing-top-k-aggregation-snowflake)
* [Dynamic Filters in Starburst](https://docs.starburst.io/latest/admin/dynamic-filtering.html),
Related items
- [x] https://github.com/apache/datafusion/issues/15037
- [x] https://github.com/apache/datafusion/issues/7955
- [x] https://github.com/apache/datafusion/issues/15513
- [x] https://github.com/apache/datafusion/issues/15534
Contributor guide
Research direction
Start by reading the linked related issues, especially 15037, 7955, 15513, and 15534, along with the Snowflake and Starburst background links. Use those items to determine the intended scope of the dynamic-filtering effort; this epic is complete only when its related work and completion criteria are defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100