apache / apache/datafusion

[DISCUSSION] Future of Dynamic Filters Sync

Open
#21,207 85 comments 3 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Overview

There are several contributors that have been actively working on dynamic filters and have expressed interest in aligning on the path forward. A few open design questions have emerged across recent PRs:

- First-class `DynamicFilter` type vs. continuing to extend `PhysicalExpr` (#20901 )
- Serialization and deduplication strategy (#20416 , #20418 )
- Cross-network propagation in distributed execution (https://github.com/datafusion-contrib/datafusion-distributed/issues/180)

We'd like to schedule one public sync call/calls to talk more about the future of dynamic filters. Please let us know availability or questions.

**Tuesday, March 31st at 11:30 AM ET**
Link: https://calendar.google.com/calendar/u/0/r/eventedit/copy/MnBzMnMyOHY3dWY2czBxZjRxMzVyMjczODAgZ2VuZS5ib3JkZWdhcmF5QGRhdGFkb2docS5jb20/Z2VuZS5ib3JkZWdhcmF5QGRhdGFkb2docS5jb20

cc @alamb @adriangb @jayshrivastava @gabotechs @liamzwbao @stuhood @geoffreyclaude @NGA-TRAN

Contributor guide

Open the contributing guide

Research direction

Start by reading PRs #20901, #20416, and #20418, along with the linked distributed-execution issue #180 and this discussion thread. Review the proposed DynamicFilter type, serialization and deduplication questions, and cross-network propagation concerns. Done means the contributors agree on and document a clear path forward.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.