[EPIC]: Make `PiecewiseMergeJoin` work in Datafusion
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
I will organize the following tasks that will be done in separate pull requests as #16660 got a little large and confusing.
The following is to be done roughly in order:
- #17482
- #18392
- Improve metrics accounting
- Mark join support
- Serialization
Smaller Tasks Along the Way:
- Move predicate extraction to Optimizer rule
- Support leftover predicates
- Support swap inputs
- https://github.com/apache/datafusion/issues/18221
Optimizations for semi/anti joins
- look into inner joins having the smaller side on the buffer side.
- Instead do no sort on buffer side, and just run a filter with the min/max value calculated from the stream side
The classic join and existence join will have their own stream implementation as the logic is completely different. Nice little suggestion from @2010YOUY01 😄
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.