[VL][SPARK-4.0][1.6.0] JVM crash (SIGSEGV) when enabling GlutenPlugin on Spark 4.0.1 with `gluten-velox-bundle-spark4.0_2.13-linux_amd64-1.6.0.jar`
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
## Bug description
When running Spark SQL with Gluten enabled on Spark 4.0.1, JVM crashes with a fatal error (`SIGSEGV`) shortly after loading `libgluten.so` and `libvelox.so`.
### Expected behavior
Query should run successfully (or fallback gracefully), without JVM crash.
### Actual behavior
Driver JVM crashes with `SIGSEGV` in `libjvm.so` during JNI method lookup path (`jni_GetMethodID`), and `hs_err_pid*.log` contains:
- `NoClassDefFoundError: Lorg/apache/gluten/memory/listener/ReservationListener;`
- native libraries loaded from Gluten bundle (`libgluten.so`, `libvelox.so`)
- process exits due to fatal JVM error
This issue report was drafted with assistance from AI.
### Gluten version
main branch
### Spark version
spark-4.0.x
### Spark configurations
Sanitized key configs used in reproduction:
--master local[80]
--class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
--jars //gluten-velox-bundle-spark4.0_2.13-linux_amd64-1.6.0.jar
--conf spark.plugins=org.apache.gluten.GlutenPlugin
--conf spark.memory.offHeap.enabled=true
--conf spark.memory.offHeap.size=20g
--conf spark.driver.memory=24g
--conf spark.sql.shuffle.partitions=80
--conf spark.sql.crossJoin.enabled=true
--conf spark.sql.legacy.timeParserPolicy=LEGACY
--conf spark.sql.ansi.enabled=false
--conf spark.gluten.sql.columnar.forceShuffledHashJoin=true
--conf spark.sql.warehouse.dir=//warehouse
spark.driver.extraJavaOptions:
-Dderby.system.home=/
-XX:+UseG1GC
### System information
JVM args (sanitized, key ones):
JDK: Temurin 17.0.19+10
--add-modules=jdk.incubator.vector
multiple --add-opens=...=ALL-UNNAMED
-Djdk.reflect.useDirectMethodHandle=false
-Dio.netty.tryReflectionSetAccessible=true
System information
Sanitized environment details (from hs_err):
OS: Alibaba Cloud Linux 3
Kernel: 5.10.134-18.al8.x86_64
Arch: x86_64
CPU: Intel(R) Xeon(R) Platinum 8369B, 80 cores
Memory: ~114 GB
glibc: 2.32
Java: OpenJDK Temurin 17.0.19+10 (linux-amd64)
(If needed, I can run dev/info.sh and provide a redacted output.)
### Relevant logs
```bash
# A fatal error has been detected by the Java Runtime Environment:
# SIGSEGV (0xb)
# JRE version: OpenJDK Runtime Environment Temurin-17.0.19+10
# Java VM: OpenJDK 64-Bit Server VM ... linux-amd64
# Problematic frame:
# V [libjvm.so+0x28cda0] ... oop_access_barrier(void*)+0x0
Internal exceptions (20 events):
Event: 14.409 Thread ... Exception
thrown [src/hotspot/share/classfile/systemDictionary.cpp, line 245]
Event: 3.960 Loaded shared library .../linux/amd64/libgluten.so
Event: 4.996 Loaded shared library .../linux/amd64/libvelox.so
java_command: org.apache.spark.deploy.SparkSubmit ...
--jars //gluten-velox-bundle-spark4.0_2.13-linux_amd64-1.6.0.jar ...
Reproduction notes
Same machine can run older Spark 3.3 + Gluten bundle without this crash.
Crash is observed in Spark 4.0.1 + Gluten 1.6.0 (Spark 4.0 / Scala 2.13 bundle).
```
Contributor guide
Research direction
Start by reproducing the crash through SparkSQLCLIDriver with the Spark 4.0.1 Gluten bundle and inspect the generated hs_err_pid*.log, especially the missing ReservationListener and native-library entries. Done means Spark SQL runs without a JVM SIGSEGV or falls back gracefully under the reported configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, sql
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100