[VL] Setting higher value of Broadcast Hash Join threshold is degrading the performance of Spark with Gluten
Open
bug
triage
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
Setup: Running TPCDS benchmark on spark with gluten.
With increasing broadcast hash join threshold, the performance of TPCDS benchmark degrades (config: `spark.sql.autoBroadcastJoinThreshold`).
Config-Value|TPCDS Total Time Taken
-|-
10MB|795sec
50MB|870sec
100MB|890sec
We have jobs with BHJ threshold set to 200MB in vanilla spark. Such jobs are not showing any gains with Spark+Gluten.
### Spark version
Spark-3.5.x
### Spark configurations
`spark.sql.autoBroadcastJoinThreshold`
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.