apache / apache/gluten

[VL] TPCDS Performance drop after new operator "VeloxAppendBatches"

Open
#6,694 16 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

We have observed performance drop in TPCDS runs after the patch #6009.

Top regressing Queries
--------------------------------------------
QueryId New runtime Previous Runtime
query64 50712 22841
query24a 44883 27452
query24b 45003 28742

When we disabled the feature using ` "spark.gluten.sql.columnar.backend.velox.coalesceBatchesBeforeShuffle": "false"`. We see the same runtime as previous runs.

We are using azure cluster and reading data from remote storage account. The regression is seen in `VeloxAppendBatches` where in some instances, it is taking a lot of time

Below are the plan snippets from query64

![image](https://github.com/user-attachments/assets/6d2b8f0c-2f30-4d71-988d-31f181a6b54a)

![image](https://github.com/user-attachments/assets/abb4121c-b659-4f4a-9a12-eb4fe29c5c14)

### Spark version

Spark-3.4.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by comparing patch #6009 with the VeloxAppendBatches execution for TPCDS queries query64, query24a, and query24b on the Azure remote-storage setup. Reproduce the runs with spark.gluten.sql.columnar.backend.velox.coalesceBatchesBeforeShuffle enabled and disabled, then identify why the enabled path regresses. Done means the affected queries return to the previous runtimes without disabling the feature.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, scala, spark
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.