[DISCUSSION] Future of Dynamic Filters Sync
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
## Overview
There are several contributors that have been actively working on dynamic filters and have expressed interest in aligning on the path forward. A few open design questions have emerged across recent PRs:
- First-class `DynamicFilter` type vs. continuing to extend `PhysicalExpr` (#20901 )
- Serialization and deduplication strategy (#20416 , #20418 )
- Cross-network propagation in distributed execution (https://github.com/datafusion-contrib/datafusion-distributed/issues/180)
We'd like to schedule one public sync call/calls to talk more about the future of dynamic filters. Please let us know availability or questions.
**Tuesday, March 31st at 11:30 AM ET**
Link: https://calendar.google.com/calendar/u/0/r/eventedit/copy/MnBzMnMyOHY3dWY2czBxZjRxMzVyMjczODAgZ2VuZS5ib3JkZWdhcmF5QGRhdGFkb2docS5jb20/Z2VuZS5ib3JkZWdhcmF5QGRhdGFkb2docS5jb20
cc @alamb @adriangb @jayshrivastava @gabotechs @liamzwbao @stuhood @geoffreyclaude @NGA-TRAN
Contributor guide
Research direction
Start by reading PRs #20901, #20416, and #20418, along with the linked distributed-execution issue #180 and this discussion thread. Review the proposed DynamicFilter type, serialization and deduplication questions, and cross-network propagation concerns. Done means the contributors agree on and document a clear path forward.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100