[Improvement] The execution time of spark connector is 4 times that of native spark3.3.2 when running tpcds sql99
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### What would you like to be improved?
A comparative test using tpcds 1000G data found that the execution time of spark connector is 4 times that of native spark 3.3.2.
Comparing the physical execution plans of the two, the Gravitino Spark Connector does not use the predicate pushdown and dynamic partition pruning optimization strategies.
[spark-gravitino-tpcds性能对比测试.xlsx](https://github.com/user-attachments/files/19861742/spark-gravitino-tpcds.xlsx)
[tpcds-query1-gravitino执行计划.txt](https://github.com/user-attachments/files/19861741/tpcds-query1-gravitino.txt)
[tpcds-query1-spark执行计划.txt](https://github.com/user-attachments/files/19861740/tpcds-query1-spark.txt)
### How should we improve?
_No response_
Contributor guide
Research direction
Start by comparing tpcds-query1-gravitino执行计划.txt with tpcds-query1-spark执行计划.txt and reviewing the attached TPCDS comparison workbook. Identify where predicate pushdown and dynamic partition pruning are absent from the Gravitino Spark Connector plan; done means both optimizations are used and the execution-time gap is remeasured.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100