datafusion-contrib / datafusion-contrib/datafusion-distributed
[dynamic filtering] remove network boundary anchors
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 139
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 35
Description
In https://github.com/datafusion-contrib/datafusion-distributed/pull/634, we added dynamic_filter_anchors.
In df55, HashJoinExec checks the plan below during execution for a dynamic filter consumer. These consumers disappear when we chop up the plan using network boundaries. The anchors are false references used to trick the dynamic filter producers into thinking consumers exist below.
This is actually fixed in https://github.com/apache/datafusion/pull/24601 by making producers decide during planning to produce filter updates.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the dynamic_filter_anchors implementation and its callers in datafusion-distributed. Read datafusion-contrib/datafusion-distributed#634 and compare the planning change in apache/datafusion#24601 to identify what is now obsolete. Done means the anchor mechanism is removed without leaving references or breaking the distributed dynamic-filter flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100