apache / apache/gluten

malloc memeory error in running tpcds

Open
#9,266 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

running tpcds queries from q1-q100 conitinous with one session ,when arrives at q3 throw exception and terminal
the error logs:
it seems malloc() error , gluten:1.3.0-velox spark 3.5.2 jdk: 17.0.6+9.
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0x7) at pc=0x0000000000000001, pid=968528, tid=1012570
#
# JRE version: OpenJDK Runtime Environment (Red_Hat-17.0.6.0.9-0.3.ea.el8) (17.0.6+9) (build 17.0.6-ea+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM (Red_Hat-17.0.6.0.9-0.3.ea.el8) (17.0.6-ea+9-LTS, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Problematic frame:
# C 0x0000000000000001
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to /opt/modules/tpcds/core.968528)
#
# An error report file with more information is saved as:
# /opt/modules/tpcds/hs_err_pid968528.log
malloc(): invalid size (unsorted)
./runtpcds_gluten.sh: line 68: 968528 Aborted (core dumped) ${SPARK_HOME}/bin/spark-submit --name "tpcds" --class com.databricks.spark.sql.perf.tpcds.RunTPCDS --master yarn --deploy-mode client --num-executors 25 --executor-cores 5 --queue queue1 --conf spark.driver.memory=4g --conf spark.executor.memory=2g --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=7g --conf spark.sql.warehouse.dir=/user/root/warehouse --conf spark.sql.catalogImplementation=hive --conf spark.sql.shuffle.partitions=2000 --conf spark.network.timeout=1200s --conf spark.speculation=true --conf spark.speculation.interval=1000 --conf spark.speculation.quantile=0.75 --conf spark.speculation.multiplier=1.5 --conf spark.memory.fraction=0.75 --conf spark.storage.storageFraction=0.4 --conf spark.sql.adaptive.enabled=true --conf spark.dynamicAllocation.enabled=false --conf spark.gluten.enabled=true --conf spark.gluten.loadLibFromJar=true --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.gluten.sql.columnar.backend.lib=velox --conf spark.gluten.sql.columnar.backend.velox.BHJOptimizeEnabled=true --conf spark.driver.extraClassPath=/opt/modules/gluten-1.3.0/jar/* --conf spark.executor.extraClassPath=/opt/modules/gluten-1.3.0/jar/* --conf spaspark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" --conf spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" --conf spark.io.compression.codec=lz4 --conf spark.sql.adaptive.localShuffleReader.enabled=true --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=133M --conf spark.gluten.sql.columnar.backend.velox.IOThreads=700 --conf spark.sql.cbo.enabled=true --conf spark.sql.cbo.joinReorder.enabled=true --conf spark.sql.cbo.planStats.enabled=true --conf spark.sql.cbo.starSchemaDetection=true --conf spark.sql.cbo.joinReorder.card.weight=0.6 --conf spark.sql.optimizer.runtime.bloomFilter.enabled=true --conf spark.sql.optimizer.runtimeFilter.semiJoinReduction.enabled=true --jars /opt/modules/gluten-1.3.0/jar/gluten-velox-bundle-spark3.5_2.12-centos_8_aarch_64-1.3.0.jar,/opt/modules/gluten-1.3.0/jar/gluten-thirdparty-lib-centos-8-aarch64.jar /opt/modules/spark-sql-perf/spark-sql-perf-assembly-0.5.1-SNAPSHOT.jar --scaleFactor 1000 --location /home/tpcds-performance-data --format parquet --dbPrefix tpcds1000g_ -i 1 -q $queries
```

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Reproduce the continuous q1–q100 TPC-DS run and inspect q3, using runtpcds_gluten.sh at line 68 and the generated hs_err_pid968528.log. Check whether the SIGBUS, invalid malloc size, and core dump recur under the listed Velox and Spark configuration; done means identifying a reproducible failure cause or the missing diagnostic needed to proceed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, linux
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.