apache / apache/gluten

[VL] Performance degraded when running with centOS9 build

Open
#6,943 3 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

We are currently working on integrating Gluten into our WatsonX.Data's Spark environment. However, after enabling Gluten and running the TPCH benchmark at the 100G scale, we are not observing the performance improvements as claimed in the Gluten repository. Specifically, we are seeing only a 10-12% improvement, whereas a 2x improvement is expected.

Here are the details of our environment:
1. Gluten was built on CentOS 9.
2. The built jar and shared libraries are being utilized on Docker images based on UBI-9.
3. Our Spark application is running with 2 executors, each configured with 6 cores and 24GB of memory.
4. We are processing TPCH data at a 100G scale, with data stored in Iceberg format.
5. We are using Java 17.

We have experimented with various configurations, but the performance gain has not exceeded 10-12% across all 22 queries. We have attached a graph showing the performance comparison between runs with and without Gluten.

image

Adding spark events for single query - Q6
[f2b74f64-bdfe-42ba-a6f7-ad81028cb2d7_events.zip](https://github.com/user-attachments/files/16675593/f2b74f64-bdfe-42ba-a6f7-ad81028cb2d7_events.zip)
cc: @deepashreeraghu @majetideepak

### Spark version

Spark-3.4.x

### Spark configurations

```
### spark configs for driver and executor
"spark.executor.cores": "6",
"spark.executor.memory": "24G",
"spark.driver.cores": "6",
"spark.driver.memory": "24G",
"spark.driver.extraClassPath": "/opt/ibm/spark/external-jars/gluten-velox-bundle-spark3.4_2.12-centos_9_x86_64-1.2.0-SNAPSHOT.jar",
"spark.executor.extraClassPath": "/opt/ibm/spark/external-jars/gluten-velox-bundle-spark3.4_2.12-centos_9_x86_64-1.2.0-SNAPSHOT.jar",
"spark.hive.metastore.uris": "thrift://",
"spark.sql.defaultCatalog": "lakehouse",
"spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
"spark.sql.catalog.lakehouse": "org.apache.iceberg.spark.SparkCatalog",
"spark.sql.catalog.lakehouse.type": "hive",
"spark.sql.iceberg.vectorization.enabled": "false",
"spark.hive.metastore.client.auth.mode": "PLAIN",
"spark.hive.metastore.client.plain.username": "",
"spark.hive.metastore.client.plain.password": "",
"spark.hive.metastore.use.SSL": "true",
"spark.hive.metastore.truststore.type": "JKS",
"spark.hive.metastore.truststore.path": "file:///opt/ibm/jdk/lib/security/cacerts",
"spark.hive.metastore.truststore.password": "changeit",

### Main Gluten configs
"spark.gluten.enabled": "true",
"spark.plugins": "org.apache.gluten.GlutenPlugin",
"spark.shuffle.manager": "org.apache.spark.shuffle.sort.ColumnarShuffleManager",
"spark.gluten.loadLibFromJar": "true",
"spark.gluten.sql.columnar.forceShuffledHashJoin": "true",
"spark.gluten.sql.columnar.backend.lib": "velox",

### Java-related updates
"spark.driver.extraJavaOptions": "-Dio.netty.tryReflectionSetAccessible=true -XX:MaxDirectMemorySize=1G -Djdk.nio.maxCachedBufferSize=262144",
"spark.executor.extraJavaOptions": "-Dio.netty.tryReflectionSetAccessible=true -XX:MaxDirectMemorySize=1G -Djdk.nio.maxCachedBufferSize=262144",

### Fallback-related
"spark.gluten.sql.columnar.joinOptimizationLevel": "18",
"spark.gluten.sql.columnar.physicalJoinOptimizeEnable": "true",
"spark.gluten.sql.columnar.physicalJoinOptimizationLevel": "18",
"spark.gluten.sql.columnar.logicalJoinOptimizeEnable": "true",
"spark.gluten.sql.columnar.logicalJoinOptimizationLevel": "19",
"spark.gluten.sql.columnar.fallback.expressions.threshold": "2",

### Memory-related
"spark.memory.offHeap.enabled": "true",
"spark.executor.memoryOverheadFactor": "75",
"spark.memory.offHeap.size": "18g",

#### AQE-related
"spark.sql.adaptive.enabled": "true",
"spark.gluten.sql.columnar.shuffle.writeEOS": "false",
"spark.gluten.sql.columnar.backend.ch.shuffle.hash.algorithm": "sparkMurmurHash3_32",

### Shuffle and Compression-related
"spark.shuffle.compress": "true",
"spark.gluten.sql.columnar.shuffle.compressionMode": "buffer",
"spark.sql.optimizer.runtime.bloomFilter.enabled": "true",
"spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold": "1KB",
"spark.gluten.sql.columnar.force.hashagg": "false",
```

Ran with 2 executors of (6*24G)

### System information

_No response_

### Relevant logs

We observed that one stage took significantly longer to complete. Could you please investigate the cause of the delay?
![image](https://github.com/user-attachments/assets/486c954f-6d65-4993-9b3e-251df2280e50)

![image](https://github.com/user-attachments/assets/4072c5ad-2f8b-4e48-ab1a-55bdf646770c)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.