apache / apache/gluten

[VL] Use of ColumnarShuffleManager leads to crash in libgluten.so

Open
#3,868 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

Enabling the ColumnarShuffleManager leads to a crash, when performing a `count("*")` even on a small dataset.

### Expected Behavior

`spark.range(0,100).select(count("*")).show` should run successfully and print out `100`.

### Actual behavior

It crashes in the libgluten.so.

### Reproduction

#### Create the docker image

```dockerfile
FROM ubuntu:22.04

RUN apt-get update && \
apt-get install -y wget tar openjdk-8-jre

RUN mkdir -p /opt/spark && \
wget \
--quiet \
--output-document=- \
"https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz" | \
tar -xz -C /opt/spark --strip-components=1

RUN wget \
--quiet \
--output-document=/opt/spark/jars/gluten-velox-bundle-spark3.3_2.12-ubuntu_22.04-1.0.0.jar \
"https://github.com/oap-project/gluten/releases/download/v1.0.0/gluten-velox-bundle-spark3.3_2.12-ubuntu_22.04-1.0.0.jar"
```

#### Build container

```bash
docker build --tag gluten-shuffle-crash .
```

#### Start container
```bash
docker run --rm -ti -e JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 gluten-shuffle-crash bash
```

#### Start spark shell
```bash
/opt/spark/bin/spark-shell \
--name run_gluten \
--conf spark.plugins=io.glutenproject.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g \
--conf spark.gluten.sql.columnar.backend.lib=velox \
--conf spark.sql.adaptive.enabled=false \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager

```

#### Perform a simple shuffle
```scala
spark.range(0,100).select(count("*")).show
```

This leads to a crash in libgluten.so. Similar crashes also happen with other prebuilt jars for centos 7 and when building the main branch for ubuntu 22.04.

When the option `--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager` is removed, the result is computed correctly and without any error.

### Spark version

Spark-3.3.x

### Spark configurations

```sh
/opt/spark/bin/spark-shell \
--name run_gluten \
--conf spark.plugins=io.glutenproject.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g \
--conf spark.gluten.sql.columnar.backend.lib=velox \
--conf spark.sql.adaptive.enabled=false \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
```

### System information

Docker container with Centos 7 (docker.io/library/centos:7@sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4)

### Relevant logs

```bash
root@74caa89bde45:/# /opt/spark/bin/spark-shell \
--name run_gluten \
--conf spark.plugins=io.glutenproject.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g \
--conf spark.gluten.sql.columnar.backend.lib=velox \
--conf spark.sql.adaptive.enabled=false \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/11/28 12:39:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://74caa89bde45:4040
Spark context available as 'sc' (master = local[*], app id = local-1701175203192).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.3.1
/_/

Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 1.8.0_382)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.range(1,100).select(count("*")).show
#Stage 0:> (0 + 16) / 16]
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x00007f85d92f60fb, pid=20, tid=0x00007f85d01ff640
#
# JRE version: OpenJDK Runtime Environment (8.0_382-b05) (build 1.8.0_382-8u382-ga-1~22.04.1-b05)
# Java VM: OpenJDK 64-Bit Server VM (25.382-b05 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libgluten.so+0x26e0fb] Java_io_glutenproject_vectorized_ShuffleReaderJniWrapper_make+0x29b
#
# Core dump written. Default location: //core or core.20
#
# An error report file with more information is saved as:
# //hs_err_pid20.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
/opt/spark/bin/spark-shell: line 47: 20 Aborted (core dumped) "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$@"
```

The whole core dump can be found here: [hs_err_pid445.log](https://github.com/oap-project/gluten/files/13488220/hs_err_pid445.log)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.