apache / apache/gluten

Performance Discrepancy: Velox Accelerates Spark on x86 but Not on ARM

Open
#8,514 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

I conducted a performance comparison of Spark with and without Velox on both x86 and ARM architectures using the TPC-H Query 5. The results indicate that while Velox enhances performance on x86, it does not yield the same improvement on ARM.

**Observed Performance Metrics:**
**x86 Environment:**
Without Velox: 109.37 seconds
With Velox: 44.04 seconds
Improvement: ~59.7% reduction in execution time
**ARM Environment:**
Without Velox: 102.82 seconds
With Velox: 102.96 seconds
Improvement: No significant improvement observed

### Spark version

Spark-3.5.x

### Spark configurations

--conf spark.memory.offHeap.enabled=true
--conf spark.memory.offHeap.size=12g
--conf spark.gluten.sql.columnar.forceShuffledHashJoin=true
--conf spark.driver.memory=4G
--conf spark.executor.instances=1
--conf spark.executor.memory=30G
--conf spark.executor.cores=16
--conf spark.sql.shuffle.partitions=32
--conf spark.executor.memoryOverhead=2g
--conf spark.driver.maxResultSize=2g

### System information

Configuration Details:
Hardware Details:
x86 Environment:
Instance Type: m6i.4xlarge
CPU: Intel Xeon (Ice Lake)
Cores: 16 vCPUs
RAM: 64 GB
ARM Environment:
Instance Type: m7g.4xlarge
CPU: AWS Graviton3
Cores: 16 vCPUs
RAM: 64 GB

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing TPC-H Query 5 with Spark 3.5.x using the listed configurations on the m6i.4xlarge and m7g.4xlarge environments, then compare execution behavior with and without Velox. Done means the ARM performance discrepancy is explained and, if a defect is confirmed, the result is validated by a benchmark showing the expected improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
data-engineering, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.