apache / apache/datafusion-comet
Improve RewriteJoin logic to calculate hash table size
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
This is a follow on issue based on discussions in https://github.com/apache/datafusion-comet/pull/1424.
When choosing the smaller side of a join to use for the build-side, we just use the total table size based on the `sizeInBytes` that was computed in a completed query stage.
We can make some improvements to this approach:
- Calculate the resulting hash table size based on the join keys and the columns from the table that will be used in the join. We can compute size based on rowCount * sum(estimated size of each column).
- In cases where the input is now a completed query stage, we can look at the HadoopFsRelation contained by the LogicalRelation. From this, we can can sizeInBytes and infer a row count based on this and the estimated schema size
### Describe the potential solution
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.