[EPIC] Sort-merge join (SMJ) execution improvements
Open
EPIC
performance
physical-plan
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
I have a number of things I'd like to keep improving for SMJ, particularly because DataFusion Comet still relies on it heavily for larger-than-memory joins.
- [x] Specialized stream for semi/anti SMJ #20806
- [x] Remove semi/anti logic from existing SMJ stream, and optimize simplified logic #21184
- [x] move Mark join logic to the Semi and Anti stream, since the logic is similar #21184
- [x] More use of Arrow-rs `DynComparator` where appropriate to reduce per-row type matching #21484
- [x] Replace `join_filter_not_matched_map` `HashMap` with `Vec` #21517
Contributor guide
Assessment
This issue has not been assessed yet.