datafusion-contrib / datafusion-contrib/datafusion-distributed
[AQE] Reorder join trees using runtime cardinality statistics
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 139
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 35
Description
Part of #522.
Use statistics collected by runtime sampling for swapping JOIN orders during dynamic planning.
The plan mutation should likely happen right in this point:
At that point, all the NetworkBoundary nodes below have already collected runtime stats, so we have a very good intuition about the incoming data.
It would be cool to rely on upstream rules for join selection, but I'm not sure if it's possible.
For more info about the background of this issue, see #522.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at src/coordinator/prepare_dynamic_plan.rs lines 71-75 and inspect how NetworkBoundary nodes expose the runtime statistics collected before that point. Read issue #522 for the background and investigate whether upstream join-selection rules can be reused. Done means dynamic planning can use sampled cardinalities to reorder JOIN trees, with the behavior validated by the project's relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100