malloc memeory error in running tpcds
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
running tpcds queries from q1-q100 conitinous with one session ,when arrives at q3 throw exception and terminal
the error logs:
it seems malloc() error , gluten:1.3.0-velox spark 3.5.2 jdk: 17.0.6+9.
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0x7) at pc=0x0000000000000001, pid=968528, tid=1012570
#
# JRE version: OpenJDK Runtime Environment (Red_Hat-17.0.6.0.9-0.3.ea.el8) (17.0.6+9) (build 17.0.6-ea+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM (Red_Hat-17.0.6.0.9-0.3.ea.el8) (17.0.6-ea+9-LTS, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Problematic frame:
# C 0x0000000000000001
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to /opt/modules/tpcds/core.968528)
#
# An error report file with more information is saved as:
# /opt/modules/tpcds/hs_err_pid968528.log
malloc(): invalid size (unsorted)
./runtpcds_gluten.sh: line 68: 968528 Aborted (core dumped) ${SPARK_HOME}/bin/spark-submit --name "tpcds" --class com.databricks.spark.sql.perf.tpcds.RunTPCDS --master yarn --deploy-mode client --num-executors 25 --executor-cores 5 --queue queue1 --conf spark.driver.memory=4g --conf spark.executor.memory=2g --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=7g --conf spark.sql.warehouse.dir=/user/root/warehouse --conf spark.sql.catalogImplementation=hive --conf spark.sql.shuffle.partitions=2000 --conf spark.network.timeout=1200s --conf spark.speculation=true --conf spark.speculation.interval=1000 --conf spark.speculation.quantile=0.75 --conf spark.speculation.multiplier=1.5 --conf spark.memory.fraction=0.75 --conf spark.storage.storageFraction=0.4 --conf spark.sql.adaptive.enabled=true --conf spark.dynamicAllocation.enabled=false --conf spark.gluten.enabled=true --conf spark.gluten.loadLibFromJar=true --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.gluten.sql.columnar.backend.lib=velox --conf spark.gluten.sql.columnar.backend.velox.BHJOptimizeEnabled=true --conf spark.driver.extraClassPath=/opt/modules/gluten-1.3.0/jar/* --conf spark.executor.extraClassPath=/opt/modules/gluten-1.3.0/jar/* --conf spaspark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" --conf spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" --conf spark.io.compression.codec=lz4 --conf spark.sql.adaptive.localShuffleReader.enabled=true --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=133M --conf spark.gluten.sql.columnar.backend.velox.IOThreads=700 --conf spark.sql.cbo.enabled=true --conf spark.sql.cbo.joinReorder.enabled=true --conf spark.sql.cbo.planStats.enabled=true --conf spark.sql.cbo.starSchemaDetection=true --conf spark.sql.cbo.joinReorder.card.weight=0.6 --conf spark.sql.optimizer.runtime.bloomFilter.enabled=true --conf spark.sql.optimizer.runtimeFilter.semiJoinReduction.enabled=true --jars /opt/modules/gluten-1.3.0/jar/gluten-velox-bundle-spark3.5_2.12-centos_8_aarch_64-1.3.0.jar,/opt/modules/gluten-1.3.0/jar/gluten-thirdparty-lib-centos-8-aarch64.jar /opt/modules/spark-sql-perf/spark-sql-perf-assembly-0.5.1-SNAPSHOT.jar --scaleFactor 1000 --location /home/tpcds-performance-data --format parquet --dbPrefix tpcds1000g_ -i 1 -q $queries
```
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Reproduce the continuous q1–q100 TPC-DS run and inspect q3, using runtpcds_gluten.sh at line 68 and the generated hs_err_pid968528.log. Check whether the SIGBUS, invalid malloc size, and core dump recur under the listed Velox and Spark configuration; done means identifying a reproducible failure cause or the missing diagnostic needed to proceed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, linux
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100