datafusion-contrib / datafusion-contrib/datafusion-distributed
[dynamic filtering] 3. benchmark / tests for distributed dynamic filtering local case
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 139
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 35
Description
Assuming https://github.com/apache/datafusion/pull/21055 is active, distributed dynamic filtering should work automatically when the filter producer node is local to the consumer. We should benchmark some queries to see if this is actually active. Inspecting the output_rows on data sources is a good litmus test.
It would be ideal if we had a quick testing framework to verify that dynamic filtering is working.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the behavior described in Apache DataFusion PR 21055 and identify how local producer-to-consumer dynamic filtering is exercised. Benchmark representative distributed queries and inspect output_rows on their data sources. Done means the benchmarks verify whether local dynamic filtering is active and provide a quick repeatable way to test it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100