apache / apache/gluten

[VL] libgluten.so crash while building gluten velox

Open
#6,088 4 comments 0 reactions 0 assignees View on GitHub
BUILD
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Problem description

**Backend**
Velox

**Bug description**
I built velox gluten according to the requirements of the official website, but encountered an error of 'C [libgluten. so+0x317353]' during startup

**Reproduction**
**Create the docker container**
```
docker pull ubuntu:22.04
docker run -itd --name ubuntu2204 ubuntu:22.04 /bin/bash
docker attach ubuntu2204

apt-get update
apt install software-properties-common
apt install maven build-essential cmake libssl-dev libre2-dev libcurl4-openssl-dev clang lldb lld libz-dev git ninja-build uuid-dev autoconf-archive curl zip unzip tar pkg-config bison libtool flex vim
apt install sudo
apt purge libjemalloc-dev libjemalloc2 librust-jemalloc-sys-dev
apt install -y openjdk-8-jdk
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export PATH=$JAVA_HOME/bin:$PATH

/spark/spark/bin/spark-shell --name run_gluten \
--master local --deploy-mode client \
--conf spark.plugins=io.glutenproject.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g \
--jars /spark/gluten-velox-bundle-spark3.2_2.12-1.1.1.jar \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
```

**error message**
```
root@8c55a15ec775:/spark/spark# /spark/spark/bin/spark-shell --name run_gluten \
--master local --deploy-mode client \
--conf spark.plugins=io.glutenproject.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g \
--jars /spark/gluten-velox-bundle-spark3.2_2.12-1.1.1.jar \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
24/06/14 01:39:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/gluten-7cca4e91-463a-4240-b9e4-36f67b147160/jni/11f04b97-1abd-4656-aeeb-0bedfe813394/gluten-984801035636929938/libvelox.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x00007f2841964353, pid=42017, tid=0x00007f28da678640
#
# JRE version: OpenJDK Runtime Environment (8.0_412-b08) (build 1.8.0_412-8u412-ga-1~22.04.1-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.412-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libgluten.so+0x317353] gluten::Runtime::registerFactory(std::string const&, std::function, std::equal_to, std::allocator > > const&)>)+0x23
#
# Core dump written. Default location: /spark/spark/core or core.42017
#
# An error report file with more information is saved as:
# /spark/spark/hs_err_pid42017.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
/spark/spark/bin/spark-shell: line 47: 42017 Aborted (core dumped) "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$@"
```

What's the problem?

### System information

server: centos7
container: ubuntu22.04

### CMake log

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the provided Ubuntu 22.04 container and Spark shell command, then inspect /spark/spark/hs_err_pid42017.log and the generated core dump. Start at the reported libgluten.so frame in gluten::Runtime::registerFactory; done means the native startup path no longer aborts with SIGILL and the cause is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, docker, java, scala
Domain
backend, build-system, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.