datafusion-contrib / datafusion-contrib/datafusion-distributed
[dynamic filtering] merge and forward updates for non complete dynamic filters
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 139
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 35
Description
https://github.com/datafusion-contrib/datafusion-distributed/pull/636 and https://github.com/datafusion-contrib/datafusion-distributed/pull/637 implement merging for dynamic filters which are marked as complete, such as ones in joins.
This issue tracks update forwarding for AggregateExec / SortExec which produce incremental updates and can continuously be forwarded to workers.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading pull requests 636 and 637 to understand the existing merging for complete dynamic filters. Then trace AggregateExec and SortExec, which produce incremental updates, and determine how those updates reach workers. Done means non-complete dynamic filter updates are merged and continuously forwarded to workers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100