apache / apache/gluten

[VL] Setting higher value of Broadcast Hash Join threshold is degrading the performance of Spark with Gluten

Open
#7,548 5 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

Setup: Running TPCDS benchmark on spark with gluten.

With increasing broadcast hash join threshold, the performance of TPCDS benchmark degrades (config: `spark.sql.autoBroadcastJoinThreshold`).

Config-Value|TPCDS Total Time Taken
-|-
10MB|795sec
50MB|870sec
100MB|890sec

We have jobs with BHJ threshold set to 200MB in vanilla spark. Such jobs are not showing any gains with Spark+Gluten.

### Spark version

Spark-3.5.x

### Spark configurations

`spark.sql.autoBroadcastJoinThreshold`

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.