apache / apache/gravitino

[Improvement] The execution time of spark connector is 4 times that of native spark3.3.2 when running tpcds sql99

Open
#7,048 6 comments 1 reaction 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What would you like to be improved?

A comparative test using tpcds 1000G data found that the execution time of spark connector is 4 times that of native spark 3.3.2.
Comparing the physical execution plans of the two, the Gravitino Spark Connector does not use the predicate pushdown and dynamic partition pruning optimization strategies.

[spark-gravitino-tpcds性能对比测试.xlsx](https://github.com/user-attachments/files/19861742/spark-gravitino-tpcds.xlsx)
[tpcds-query1-gravitino执行计划.txt](https://github.com/user-attachments/files/19861741/tpcds-query1-gravitino.txt)
[tpcds-query1-spark执行计划.txt](https://github.com/user-attachments/files/19861740/tpcds-query1-spark.txt)

### How should we improve?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by comparing tpcds-query1-gravitino执行计划.txt with tpcds-query1-spark执行计划.txt and reviewing the attached TPCDS comparison workbook. Identify where predicate pushdown and dynamic partition pruning are absent from the Gravitino Spark Connector plan; done means both optimizations are used and the execution-time gap is remeasured.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
data-engineering, distributed-systems, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.