A tool to indicate perf gain of Gluten vs. vanilla Spark
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
In customer landing process, checking the performance gain requires re-run the queries with Gluten enabled - this usually brings high overhead as there are tens of thousands of queries per day.
Customer wants a tool to check how much perf gain Gluten can bring based on event log/query patterns, w/o need to running the queries with Gluten.
- the tool will parse the event log, and calculate & report the potential perf gain based on a pre-defined weights for the perf gain. e.g. for sort, gluten can provide 1.5x speedup. for hashagg, gluten can do 2x speedup.
- This doesn't have to be very accurate first, but the tool should provide basic information on the perf gain, like estimated 1.5x time saving. Normally the user will try to re-schedule the query can benefits from Gluten, and verify the results/perf gain in future runs
- The benefits can be on two factors: expressions and operators. We could conduct the perf gain with below approaches
1. pick several typical queries from TPC-H/DS. Compare the operators/expression performance in Gluten vs. Vanilla Spark
2. make a primitive SQL set which stress the performance on single operators/expressions, based on the pattern in production SQL queries. e.g., make a map join based on TPCDS table, and write the results with `noop` format to stress the join operator
CC @jackylee-ch @ulysses-you @zhli1142015
thanks, -yuan
Contributor guide
Assessment
This issue has not been assessed yet.