apache / apache/datafusion-comet
Wrap in CoalesceBatchesExec more operators
Open
enhancement
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Datafusion optimizer [wraps some operators](https://github.com/apache/datafusion/blob/3eceedcde3c8a1aba61f05ccc39edf4728b5728c/datafusion/physical-optimizer/src/coalesce_batches.rs#L63) which might produce multiple small batches with CoalesceBatchesExec node. In Comet, this technique is already applied to SMJ with filter.
### Describe the potential solution
Shouldn't it be extended to other plans (Filter/HashJoin)? It might be especially beneficial for workloads where sort merge join -> hash join replacement is enforced.
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.