apache / apache/datafusion-comet
Potential revamp of broadcast compression policy
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Currently, broadcast batches are compressed twice by the same codec:
1. During rdd formation on the comet side [here](https://github.com/apache/datafusion-comet/blob/8112e1acab497ca3a915d4ab3fdce4ce9e64c88a/common/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala#L207)
2. When the blocks creation phase initiated by spark [here](https://github.com/apache/spark/blob/7007e1c7ad646bfdc2a89579b2abaa2b3facc6af/core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala#L365)
It doesn't cause statistically significant improvement on the standard benchmarks (typically broadcasts are just tiny parts of the whole plan in terms of data movements), but it might hurt workloads where 100MB-1GB data chunks are explicitly targeted for being broadcast via hint or configuration.
### Describe the potential solution
So, the point is checking two assumptions:
1. The query performance is notably regressed on queries where larger data portions are broadcast.
2. The double compression brings no benefits in terms of inter-node traffic volume.
If both are true, maybe it is worth turning off/making configurable broadcast compression on the comet side.
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.