Performance Discrepancy: Velox Accelerates Spark on x86 but Not on ARM
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
I conducted a performance comparison of Spark with and without Velox on both x86 and ARM architectures using the TPC-H Query 5. The results indicate that while Velox enhances performance on x86, it does not yield the same improvement on ARM.
**Observed Performance Metrics:**
**x86 Environment:**
Without Velox: 109.37 seconds
With Velox: 44.04 seconds
Improvement: ~59.7% reduction in execution time
**ARM Environment:**
Without Velox: 102.82 seconds
With Velox: 102.96 seconds
Improvement: No significant improvement observed
### Spark version
Spark-3.5.x
### Spark configurations
--conf spark.memory.offHeap.enabled=true
--conf spark.memory.offHeap.size=12g
--conf spark.gluten.sql.columnar.forceShuffledHashJoin=true
--conf spark.driver.memory=4G
--conf spark.executor.instances=1
--conf spark.executor.memory=30G
--conf spark.executor.cores=16
--conf spark.sql.shuffle.partitions=32
--conf spark.executor.memoryOverhead=2g
--conf spark.driver.maxResultSize=2g
### System information
Configuration Details:
Hardware Details:
x86 Environment:
Instance Type: m6i.4xlarge
CPU: Intel Xeon (Ice Lake)
Cores: 16 vCPUs
RAM: 64 GB
ARM Environment:
Instance Type: m7g.4xlarge
CPU: AWS Graviton3
Cores: 16 vCPUs
RAM: 64 GB
### Relevant logs
_No response_
Contributor guide
Research direction
Start by reproducing TPC-H Query 5 with Spark 3.5.x using the listed configurations on the m6i.4xlarge and m7g.4xlarge environments, then compare execution behavior with and without Velox. Done means the ARM performance discrepancy is explained and, if a defect is confirmed, the result is validated by a benchmark showing the expected improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100