apache / apache/datafusion-comet

Potential revamp of broadcast compression policy

Open
#2,216 3 comments 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Currently, broadcast batches are compressed twice by the same codec:

1. During rdd formation on the comet side [here](https://github.com/apache/datafusion-comet/blob/8112e1acab497ca3a915d4ab3fdce4ce9e64c88a/common/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala#L207)
2. When the blocks creation phase initiated by spark [here](https://github.com/apache/spark/blob/7007e1c7ad646bfdc2a89579b2abaa2b3facc6af/core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala#L365)

It doesn't cause statistically significant improvement on the standard benchmarks (typically broadcasts are just tiny parts of the whole plan in terms of data movements), but it might hurt workloads where 100MB-1GB data chunks are explicitly targeted for being broadcast via hint or configuration.

### Describe the potential solution

So, the point is checking two assumptions:

1. The query performance is notably regressed on queries where larger data portions are broadcast.
2. The double compression brings no benefits in terms of inter-node traffic volume.

If both are true, maybe it is worth turning off/making configurable broadcast compression on the comet side.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.