apache / apache/gluten

Coredump on Kunpeng 920 CPU

Open
#11,954 2 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

I'm running Gluten 1.6.0 and 1.7.0 snapshot inside Ubuntu 22.04 container on Kunpeng 920 CPU, with JDK
to OpenJDK17. The JVM crashed if using gluten in spark-sql.

### Gluten version

main branch

### Spark version

Spark-3.5.x

### Spark configurations

bin/spark-sql \
--master local \
--executor-cores 4 \
--executor-memory 4g \
--num-executors 1 \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=12g \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--jars /mnt/nfs-workdir/spark-deps/gluten-jdk17-20260416/gluten-velox-bundle-spark3.5_2.12-linux_aarch64-1.6.0-SNAPSHOT.jar \
--conf spark.driver.extraClassPath=/mnt/nfs-workdir/spark-deps/gluten-jdk17-20260416/gluten-velox-bundle-spark3.5_2.12-linux_aarch64-1.6.0-SNAPSHOT.jar \
--conf spark.executor.extraClassPath=/mnt/nfs-workdir/spark-deps/gluten-jdk17-20260416/gluten-velox-bundle-spark3.5_2.12-linux_aarch64-1.6.0-SNAPSHOT.jar \

### System information

Kernel:
Linux BClinux 5.10.0-200.0.0.181.111.bclinux.21.10U4.aarch64 #1 SMP Wed Sep 24 17:45:37 CST 2025 aarch64 aarch64 aarch64 GNU/Linux

HostOS:
BigCloud Enterprise Linux 21.10U4 LTS

ContainerOS:
Ubuntu 22.04.5 LTS

# lscpu
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 192
On-line CPU(s) list: 0-191
Vendor ID: HiSilicon
Model name: Kunpeng-920
Model: 0
Thread(s) per core: 1
Core(s) per cluster: 48
Socket(s): -
Cluster(s): 4
Stepping: 0x1
Frequency boost: disabled
CPU max MHz: 2600.0000
CPU min MHz: 200.0000
BogoMIPS: 200.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm ssbs

### Relevant logs

```bash
spark-sql (default)> select count(*) from passwdx;
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x0000fffdb72d6f04, pid=2769, tid=3007
#
# JRE version: OpenJDK Runtime Environment (17.0.18+8) (build 17.0.18+8-Ubuntu-122.04.1)
# Java VM: OpenJDK 64-Bit Server VM (17.0.18+8-Ubuntu-122.04.1, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Problematic frame:
# C [libgluten.so+0x6a6f04] arrow::BaseMemoryPoolImpl::Allocate(long, long, unsigned char**)+0x104
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /opt/spark/core.2769)
#
# An error report file with more information is saved as:
# /opt/spark/hs_err_pid2769.log
#
# If you would like to submit a bug report, please visit:
# https://bugs.launchpad.net/ubuntu/+source/openjdk-17
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Aborted (core dumped)

-------------------

[New LWP 226]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
--Type for more, q to quit, c to continue without paging--
Core was generated by `/usr/lib/jvm/java-17-openjdk-arm64/bin/java -cp /mnt/nfs-workdir/spark-deps/glu'.
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill_implementation (threadid=281458197336352, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
44 ./nptl/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0xfffc17df1120 (LWP 521))]
(gdb)
(gdb)
(gdb)
(gdb) bt
#0 __pthread_kill_implementation (threadid=281458197336352, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1 0x0000ffff9b612054 in __pthread_kill_internal (signo=6, threadid=) at ./nptl/pthread_kill.c:78
#2 0x0000ffff9b5ca83c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3 0x0000ffff9b5b7134 in __GI_abort () at ./stdlib/abort.c:79
#4 0x0000ffff9ae04fb8 in os::abort(bool, void*, void const*) () from /usr/lib/jvm/java-17-openjdk-arm64/lib/server/libjvm.so
#5 0x0000ffff9b0bcb6c in VMError::report_and_die(int, char const*, char const*, std::__va_list, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long) ()
from /usr/lib/jvm/java-17-openjdk-arm64/lib/server/libjvm.so
#6 0x0000ffff9b0bd51c in VMError::report_and_die(Thread*, unsigned int, unsigned char*, void*, void*, char const*, ...) () from /usr/lib/jvm/java-17-openjdk-arm64/lib/server/libjvm.so
#7 0x0000ffff9af6035c in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-arm64/lib/server/libjvm.so
#8
#9 0x0000fffd976e6f04 in arrow::BaseMemoryPoolImpl::Allocate(long, long, unsigned char**) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#10 0x0000fffd976eb428 in arrow::PoolBuffer::Reserve(long) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#11 0x0000fffd976eb4e4 in arrow::PoolBuffer::Resize(long, bool) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#12 0x0000fffd976ea408 in arrow::AllocateResizableBuffer(long, long, arrow::MemoryPool*) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#13 0x0000fffd976ea93c in arrow::AllocateResizableBuffer(long, arrow::MemoryPool*) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#14 0x0000fffd97758704 in arrow::io::BufferedBase::ResetBuffer() ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#15 0x0000fffd9775694c in arrow::io::BufferedOutputStream::Create(long, arrow::MemoryPool*, std::shared_ptr) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#16 0x0000fffd975986dc in gluten::(anonymous namespace)::openFile(std::__cxx11::basic_string, std::allocator > const&, long) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#17 0x0000fffd9759b30c in gluten::LocalPartitionWriter::stop(gluten::ShuffleWriterMetrics*, long&) ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#18 0x0000fffd90be922c in gluten::VeloxHashShuffleWriter::stop() ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libvelox.so
#19 0x0000fffd9758719c in Java_org_apache_gluten_vectorized_ShuffleWriterJniWrapper_stop ()
from /tmp/gluten-1e27a60e-c7cd-4692-9bed-92a76da5d670/jni/969a2f58-bfc0-4b60-9d46-bad3521af894/gluten-3661091660157195524/linux/aarch64/libgluten.so
#20 0x0000ffff848c2af8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)
```

Contributor guide

Open the contributing guide

Research direction

Start with the reproduced bin/spark-sql command and the SIGILL stack trace, especially gluten::LocalPartitionWriter::stop, gluten::openFile, and Arrow's SystemAllocator::Allocate. Investigate the Kunpeng aarch64 failure path and validate changes by rerunning the query without a JVM crash; the payload does not identify a source file or existing test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.