apache / apache/gluten

A tool to indicate perf gain of Gluten vs. vanilla Spark

Open
#4,470 4 comments 9 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

In customer landing process, checking the performance gain requires re-run the queries with Gluten enabled - this usually brings high overhead as there are tens of thousands of queries per day.

Customer wants a tool to check how much perf gain Gluten can bring based on event log/query patterns, w/o need to running the queries with Gluten.

- the tool will parse the event log, and calculate & report the potential perf gain based on a pre-defined weights for the perf gain. e.g. for sort, gluten can provide 1.5x speedup. for hashagg, gluten can do 2x speedup.

- This doesn't have to be very accurate first, but the tool should provide basic information on the perf gain, like estimated 1.5x time saving. Normally the user will try to re-schedule the query can benefits from Gluten, and verify the results/perf gain in future runs
- The benefits can be on two factors: expressions and operators. We could conduct the perf gain with below approaches

1. pick several typical queries from TPC-H/DS. Compare the operators/expression performance in Gluten vs. Vanilla Spark
2. make a primitive SQL set which stress the performance on single operators/expressions, based on the pattern in production SQL queries. e.g., make a map join based on TPCDS table, and write the results with `noop` format to stress the join operator

CC @jackylee-ch @ulysses-you @zhli1142015

thanks, -yuan

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.