[VL][CELEBORN] Questions about the usage of Celeborn and concerns regarding the test performance results
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
I have the following two questions:
Question 1:
On the official website, I saw that the usage configuration for Celeborn is as follows:
> Celeborn: celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar
> Gluten: gluten-velox-bundle-spark3.x_2.12-xx_xx_xx-SNAPSHOT.jar, gluten-celeborn-package-xx-SNAPSHOT.jar
I compiled the Gluten 1.3.0 Release source code and obtained the gluten-velox-bundle-spark3.4_2.12-xx_xx_xx.jar file, along with the gluten-celeborn-1.3.0.jar. However, I only used the gluten-velox-bundle-spark3.4_2.12-xx_xx_xx.jar and did not place the gluten-celeborn-1.3.0.jar in the $SPARK_HOME/jars directory. Despite this, I was able to access the Celeborn service without any issues.
Upon checking the contents of the gluten-velox-bundle-spark3.4_2.12-xx_xx_xx.jar, I found that it includes Java and Scala class files from the gluten-celeborn module. Does this mean that the gluten-celeborn-1.3.0.jar is not necessary, and the official documentation on Celeborn Support usage is outdated?
Question 2:
Additionally, I tested a 1TB TPC-H dataset and observed a strange phenomenon. Although the overall performance improved when using Gluten + Rss compared to using Gluten alone, the performance of over ten out of the 22 SQL queries actually decreased. Has the official team also observed similar results?
### Gluten version
None
Contributor guide
Assessment
This issue has not been assessed yet.