apache / apache/datafusion-comet

Improve RewriteJoin logic to calculate hash table size

Open
#1,430 0 comments 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

This is a follow on issue based on discussions in https://github.com/apache/datafusion-comet/pull/1424.

When choosing the smaller side of a join to use for the build-side, we just use the total table size based on the `sizeInBytes` that was computed in a completed query stage.

We can make some improvements to this approach:

- Calculate the resulting hash table size based on the join keys and the columns from the table that will be used in the join. We can compute size based on rowCount * sum(estimated size of each column).
- In cases where the input is now a completed query stage, we can look at the HadoopFsRelation contained by the LogicalRelation. From this, we can can sizeInBytes and infer a row count based on this and the estimated schema size

### Describe the potential solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.