[VL] [ARM] Core Dump Issue for Benchmark with the TPCDS queries

### Backend

VL (Velox)

### Bug description

Hi Team ,

I am Benchmarking Gluten + Velox with the TPCDS benchmarking , I am using the HDFS as my storage, For the some Of the queries i am facing the Core dump issue like below

`````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````
A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x0000fffe24e11500, pid=115, tid=138

JRE version: OpenJDK Runtime Environment (17.0.13+12) (build 17.0.13+12-LTS)
Java VM: OpenJDK 64-Bit Server VM (17.0.13+12-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs,
g1 gc, linux-aarch64)
Problematic frame:
C 0x0000fffe24e11500

Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /tmp/hsperfdata_spark/core.115)

An error report file with more information is saved as:
/tmp/hsperfdata_spark/hs_err_pid115.log

If you would like to submit a bug report, please visit:
https://bell-sw.com/support

/opt/bitnami/spark/bin/spark-shell: line 47: 115 Aborted (core dumped) "${SPARK_HOME}"/bin/spark-submit -- class org.apache.spark.repl.Main --name "Spark shell" "$@"

``````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````

I am using a Scale Factor Of 1000 for this benchmarking , I am also using single spark node cluster for this experiment , Can u get some Help on this ??

Gluten version

Gluten-1.3

Below is other Software Versions we are using :

Hadoop: 3.4
Our Hadoop Cluster is Multi Node Cluster of 3 Machine with the combination of 1 - Arm Machine (Name Node) and 2 - X86 (Data Node)
Java : 17

### Spark version

Spark-3.5.x

### Spark configurations

--deploy-mode client
--executor-cores 124
--executor-memory 112g
--num-executors 1
--driver-memory 16g
--conf spark.driver.extraClassPath=/opt/gluten/package/target/gluten-velox-bundle-spark3.5_2.12-debian_12_aarch_64-1.3.0.jar
--conf spark.executor.extraClassPath=/opt/gluten/package/target/gluten-velox-bundle-spark3.5_2.12-debian_12_aarch_64-1.3.0.jar
--conf spark.default.parallelism=200
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
--conf spark.sql.adaptive.enabled=false
--conf spark.hadoop.input.connect.timeout=1000
--conf spark.hadoop.input.read.timeout=1000
--conf spark.hadoop.input.write.timeout=1000
--conf spark.memory.offHeap.enabled=true
--conf spark.memory.offHeap.size=784g
--conf spark.gluten.sql.columnar.backend.velox.orc.scan.enabled=true
--conf spark.plugins=org.apache.gluten.GlutenPlugin
--conf spark.hadoop.fs.defaultFS=hdfs://192.168.2.71:8020
--conf spark.driver.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true
--conf spark.executor.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true
--conf spark.executor.memoryOverhead=60g
--conf spark.sql.orc.columnarReaderBatchSize=10240
--conf spark.sql.broadcastTimeout=4800
--conf spark.driver.maxResultSize=4g
--conf spark.sql.shuffle.partitions=200

### System information

_No response_

### Relevant logs

NUMA_CORES: 0-123
start sf1000 at 2025/05/16 09:29:15
num_executors=1
executor_cores=124
partitions=200
offheap_mem: 784g
Backend : velox
onheap_mem: 112g
memory_overhead: 60g
+ cat /tools/run_tpcds/tpc.scala
+ spark-shell --name 20250516_092914 --master spark://192.168.2.39:31077 --deploy-mode client --executor-cores 124 --executor-memory 112g --num-executors 1 --driver-memory 16g --conf spark.driver.extraClassPath=/opt/gluten/package/target/gluten-velox-bundle-spark3.5_2.12-debian_12_aarch_64-1.3.0.jar --conf spark.executor.extraClassPath=/opt/gluten/package/target/gluten-velox-bundle-spark3.5_2.12-debian_12_aarch_64-1.3.0.jar --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.sql.adaptive.enabled=false --conf spark.hadoop.input.connect.timeout=1000 --conf spark.hadoop.input.read.timeout=1000 --conf spark.hadoop.input.write.timeout=1000 --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=784g --conf spark.gluten.sql.columnar.backend.velox.orc.scan.enabled=true --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.hadoop.fs.defaultFS=hdfs://192.168.2.71:8020 --conf spark.driver.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true --conf spark.executor.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true --conf spark.executor.memoryOverhead=60g --conf spark.sql.orc.columnarReaderBatchSize=10240 --conf spark.sql.broadcastTimeout=4800 --conf spark.driver.maxResultSize=4g --conf spark.sql.shuffle.partitions=200
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
25/05/16 09:29:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
25/05/16 09:29:24 WARN VeloxListenerApi: Memory overhead is set to 64424509440 which is smaller than the recommended size 252544077004. This may cause OOM.
W20250516 09:29:26.105852 390 MemoryArbitrator.cpp:84] Query memory capacity[45.00GB] is set for NOOP arbitrator which has no capacity enforcement
25/05/16 09:29:26 WARN SparkShimProvider: Spark runtime version 3.5.3 is not matched with Gluten's fully tested version 3.5.2
Spark context Web UI available at http://soft-spark-velox-kusanagi-master-0.soft-spark-velox-kusanagi-headless.default.svc.cluster.local:4040
Spark context available as 'sc' (master = spark://192.168.2.39:31077, app id = app-20250516092926-0001).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.5.3
/_/

Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 17.0.13)
Type in expressions to have them evaluated.
Type :help for more information.

scala> | | | | | | | | | | | | | | | | import org.apache.spark.sql.execution.debug._

scala> import scala.io.Source

scala> import java.io.File

scala> import java.util.Arrays

scala> import sys.process._

scala> import scala.util.{Try, Success, Failure, Random}

scala> import java.time.LocalDateTime

scala>
scala>
scala>
scala> | | | tpc: String = tpcds

scala> tpc: tpcds

scala>
scala> | | | file_ext: String = orc

scala> file_ext: orc

scala>
scala> | | | queries_path: String = /tools/run_tpcds/queries.gluten/

scala> queries_path: /tools/run_tpcds/queries.gluten/

scala>
scala> | | | target_query: String = q16.sql

scala> target_query: q16.sql

scala>
scala> | | | random_seed: Long = 0

scala> random_seed: 0

scala>
scala>
scala> | | | file_scheme: String = hdfs

scala> file_scheme : hdfs

scala>
scala>
scala>
scala> | | | | | | | | | | | | orc_file_root: String = hdfs://192.168.2.71:8020

scala> orc_file_root: hdfs://192.168.2.71:8020

scala>
scala> | | | | | | data_file_path: String = /spark-tpcds-data/sf1000

scala> data_file_path: /spark-tpcds-data/sf1000

scala>
scala> | | | sync: Boolean = false

scala> file_scheme : hdfs

scala>
scala> | | | | | | | | time: [R](block: => R)R

scala>
scala> | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
[Stage 0:> (0 + 124) / 2104]

[Stage 0:=> (46 + 125) / 2104]

[Stage 0:==> (111 + 125) / 2104]

[Stage 0:====> (175 + 125) / 2104]

[Stage 0:======> (254 + 126) / 2104]

[Stage 0:========> (333 + 127) / 2104]

[Stage 0:==========> (428 + 133) / 2104]

[Stage 0:============> (525 + 143) / 2104]

[Stage 0:===============> (631 + 127) / 2104]

[Stage 0:=================> (713 + 124) / 2104]

[Stage 0:====================> (822 + 125) / 2104]

[Stage 0:======================> (925 + 125) / 2104]

[Stage 0:========================> (1011 + 125) / 2104]

[Stage 0:===========================> (1125 + 125) / 2104]

[Stage 0:=============================> (1220 + 125) / 2104]

[Stage 0:================================> (1338 + 125) / 2104]

[Stage 0:==================================> (1443 + 126) / 2104]

[Stage 0:=====================================> (1542 + 125) / 2104]

[Stage 0:========================================> (1653 + 124) / 2104]

[Stage 0:===========================================> (1788 + 126) / 2104]

[Stage 0:==============================================> (1930 + 125) / 2104]

[Stage 0:=====================================================(2104 + 0) / 2104]

[Stage 1:======> (224 + 125) / 1837]

[Stage 1:==========> (377 + 124) / 1837]

[Stage 1:==============> (527 + 126) / 1837]

[Stage 1:===================> (685 + 124) / 1837]

[Stage 1:=======================> (843 + 125) / 1837]

[Stage 1:===========================> (1007 + 124) / 1837]

[Stage 1:================================> (1164 + 125) / 1837]

[Stage 1:=====================================> (1333 + 126) / 1837]

[Stage 1:=========================================> (1507 + 124) / 1837]

[Stage 1:==============================================> (1661 + 124) / 1837]

[Stage 3:=======> (280 + 125) / 2004]

[Stage 3:===========> (429 + 125) / 2004]

[Stage 3:===============> (582 + 125) / 2004]

[Stage 3:==================> (724 + 126) / 2004]

[Stage 3:======================> (884 + 124) / 2004]

[Stage 3:==========================> (1034 + 125) / 2004]

[Stage 3:==============================> (1191 + 125) / 2004]

[Stage 3:==================================> (1345 + 125) / 2004]

[Stage 3:======================================> (1499 + 126) / 2004]

[Stage 3:==========================================> (1660 + 125) / 2004]

[Stage 3:==============================================> (1810 + 125) / 2004]

[Stage 4:========> (312 + 126) / 1824]

[Stage 4:=============> (489 + 126) / 1824]

[Stage 4:==================> (654 + 126) / 1824]

[Stage 4:=======================> (809 + 126) / 1824]

[Stage 4:===========================> (963 + 126) / 1824]

[Stage 4:==============================> (1104 + 124) / 1824]

[Stage 4:===================================> (1258 + 124) / 1824]

[Stage 4:=======================================> (1412 + 126) / 1824]

[Stage 4:===========================================> (1571 + 126) / 1824]

[Stage 4:=================================================> (1727 + 97) / 1824]

[Stage 5:=========> (418 + 124) / 2184]

[Stage 5:=============> (563 + 124) / 2184]

[Stage 5:================> (705 + 124) / 2184]

[Stage 5:====================> (848 + 124) / 2184]

[Stage 5:=======================> (1001 + 124) / 2184]

[Stage 5:==========================> (1145 + 125) / 2184]

[Stage 5:==============================> (1285 + 124) / 2184]

[Stage 5:=================================> (1430 + 124) / 2184]

[Stage 5:====================================> (1572 + 124) / 2184]

[Stage 5:========================================> (1721 + 124) / 2184]

[Stage 5:===========================================> (1865 + 124) / 2184]

[Stage 5:==============================================> (2011 + 124) / 2184]

[Stage 6:===========> (404 + 126) / 1824]

[Stage 6:===============> (550 + 125) / 1824]

[Stage 6:====================> (702 + 124) / 1824]

[Stage 6:========================> (843 + 124) / 1824]

[Stage 6:============================> (988 + 125) / 1824]

[Stage 6:===============================> (1130 + 124) / 1824]

[Stage 6:===================================> (1274 + 124) / 1824]

[Stage 6:=======================================> (1427 + 124) / 1824]

[Stage 6:===========================================> (1566 + 124) / 1824]

[Stage 6:===============================================> (1712 + 112) / 1824]

25/05/16 09:30:06 WARN SparkStringUtils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.

scala>
scala>
scala> | | | shuffleList: [A](list: List[A], seed: Long)List[A]

scala>
scala> | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | getListOfFiles: (dir: String)List[java.io.File]

scala> fileLists: List[java.io.File] = List(/tools/run_tpcds/queries.gluten/q16.sql)

scala> | | | | | | | | | sorted: List[java.io.File] = List(/tools/run_tpcds/queries.gluten/q16.sql)

scala> queries: List[java.io.File] = List(/tools/run_tpcds/queries.gluten/q16.sql)

scala>
scala> | | | | | | | | | | | | | | | | | | | | sync disabled

scala>
scala> starttime: Long = 6903336867291395

scala>
scala>
scala> | | | | | | | | | | | | | | | | /tools/run_tpcds/queries.gluten/q16.sql
2025-05-16T09:30:09.299648959
select count(distinct cs_order_number) as `order count` ,sum(cs_ext_ship_cost) as `total shipping cost` ,sum(cs_net_profit) as `total net profit` from catalog_sales cs1 ,date_dim ,customer_address ,call_center where d_date between '1999-2-01' and (cast('1999-2-01' as date) + interval '60' day) and cs1.cs_ship_date_sk = d_date_sk and cs1.cs_ship_addr_sk = ca_address_sk and ca_state = 'IL' and cs1.cs_call_center_sk = cc_call_center_sk and cc_county in ('Williamson County','Williamson County','Williamson County','Williamson County', 'Williamson County' ) and exists (select * from catalog_sales cs2 where cs1.cs_order_number = cs2.cs_order_number and cs1.cs_warehouse_sk <> cs2.cs_warehouse_sk) and not exists(select * from catalog_returns cr1 where cs1.cs_order_number = cr1.cr_order_number) order by count(distinct cs_order_number) LIMIT 100 ;
/opt/bitnami/spark/bin/spark-shell: line 47: 367 Segmentation fault (core dumped) "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$@"
++ date '+%Y/%m/%d %H:%M:%S'
+ echo 'end sf1000 at 2025/05/16 09:30:18'
end sf1000 at 2025/05/16 09:30:18

Start by reproducing the crash with q16.sql from /tools/run_tpcds/queries.gluten/ using the reported ARM, HDFS, Spark 3.5.3, and Velox configuration. Inspect /tmp/hsperfdata_spark/hs_err_pid115.log and the generated core dump to identify the failing native component. Done means the benchmark no longer aborts under the reproducing query and the failure has a verified regression test or documented reproduction.

Written by the indexing model from the issue text.