apache / apache/gluten

[VL][CELEBORN] Questions about the usage of Celeborn and concerns regarding the test performance results

Open
#9,853 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

I have the following two questions:

Question 1:
On the official website, I saw that the usage configuration for Celeborn is as follows:

> Celeborn: celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar
> Gluten: gluten-velox-bundle-spark3.x_2.12-xx_xx_xx-SNAPSHOT.jar, gluten-celeborn-package-xx-SNAPSHOT.jar

I compiled the Gluten 1.3.0 Release source code and obtained the gluten-velox-bundle-spark3.4_2.12-xx_xx_xx.jar file, along with the gluten-celeborn-1.3.0.jar. However, I only used the gluten-velox-bundle-spark3.4_2.12-xx_xx_xx.jar and did not place the gluten-celeborn-1.3.0.jar in the $SPARK_HOME/jars directory. Despite this, I was able to access the Celeborn service without any issues.

Upon checking the contents of the gluten-velox-bundle-spark3.4_2.12-xx_xx_xx.jar, I found that it includes Java and Scala class files from the gluten-celeborn module. Does this mean that the gluten-celeborn-1.3.0.jar is not necessary, and the official documentation on Celeborn Support usage is outdated?

Question 2:
Additionally, I tested a 1TB TPC-H dataset and observed a strange phenomenon. Although the overall performance improved when using Gluten + Rss compared to using Gluten alone, the performance of over ten out of the 22 SQL queries actually decreased. Has the official team also observed similar results?

### Gluten version

None

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.