apache / apache/gluten

[VL] TPC-DS 1G q1 core dump in VeloxHashShuffleWriter::write when build type is DEBUG

Open
#7,350 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

In order to troubleshoot the bug, I compiled Velox and Gluten cpp using debug mode. However, when I tried to execute q1, it resulted in a core dump and couldn't be executed. The core dump information using jemalloc is as follows. If jemalloc is not used, it will throw an error "double free or corruption (out)".

There is no issue when compiling in release mode.

```
#0 0x00007fa8c4278005 in raise () from /lib64/libc.so.6
#1 0x00007fa8c424a894 in abort () from /lib64/libc.so.6
#2 0x00007fa8c24144d7 in os::abort(bool) [clone .cold] () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-2.0.1.1.al8.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007fa8c2dda9fa in VMError::report_and_die() () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-2.0.1.1.al8.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007fa8c2bc6caa in JVM_handle_linux_signal () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-2.0.1.1.al8.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007fa8c2bb9dac in signalHandler(int, siginfo_t*, void*) () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-2.0.1.1.al8.x86_64/jre/lib/amd64/server/libjvm.so
#6
#7 0x00007fa8c440efa9 in je_free_default () from /opt/apps/JINDOSDK/jindosdk-current/lib/native/libjemalloc.so
#8 0x00007fa86186c248 in std::pair, std::allocator > const, int>::~pair() ()
from /tmp/gluten-d8ba7835-dda0-466d-806b-27ad3064fe02/jni/5e6e9492-bdb2-4bf8-ac27-f667c412ace7/gluten-3499241409578429147/linux/amd64/libgluten.so
#9 0x00007fa861870784 in void __gnu_cxx::new_allocator, std::allocator > const, int>, true> >::destroy, std::allocator > const, int> >(std::pair, std::allocator > const, int>*) ()
from /tmp/gluten-d8ba7835-dda0-466d-806b-27ad3064fe02/jni/5e6e9492-bdb2-4bf8-ac27-f667c412ace7/gluten-3499241409578429147/linux/amd64/libgluten.so
#10 0x00007fa86186fdbf in void std::allocator_traits, std::allocator > const, int>, true> > >::destroy, std::allocator > const, int> >(std::allocator, std::allocator > const, int>, true> >&, std::pair, std::allocator > const, int>*) () from /tmp/gluten-d8ba7835-dda0-466d-806b-27ad3064fe02/jni/5e6e9492-bdb2-4bf8-ac27-f667c412ace7/gluten-3499241409578429147/linux/amd64/libgluten.so
#11 0x00007fa86186f473 in std::__detail::_Hashtable_alloc, std::allocator > const, int>, true> > >::_M_deallocate_node(std::__detail::_Hash_node, std::allocator > const, int>, true>*) ()
from /tmp/gluten-d8ba7835-dda0-466d-806b-27ad3064fe02/jni/5e6e9492-bdb2-4bf8-ac27-f667c412ace7/gluten-3499241409578429147/linux/amd64/libgluten.so
#12 0x00007fa86186e492 in std::__detail::_Hashtable_alloc, std::allocator > const, int>, true> > >::_M_deallocate_nodes(std::__detail::_Hash_node, std::allocator > const, int>, true>*) ()
from /tmp/gluten-d8ba7835-dda0-466d-806b-27ad3064fe02/jni/5e6e9492-bdb2-4bf8-ac27-f667c412ace7/gluten-3499241409578429147/linux/amd64/libgluten.so
#13 0x00007fa8618f5a76 in std::_Hashtable, std::allocator >, std::pair, std::allocator > const, int>, std::allocator, std::allocator > const, int> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::clear (
this=0x7fa8089ddf80) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/hashtable.h:2500
#14 0x00007fa8618f3354 in std::_Hashtable, std::allocator >, std::pair, std::allocator > const, int>, std::allocator, std::allocator > const, int> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::~_Hashtable (
this=0x7fa8089ddf80, __in_chrg=) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/hashtable.h:1584
#15 0x00007fa8618f1948 in std::unordered_multimap, std::allocator >, int, std::hash, std::allocator > >, std::equal_to, std::allocator > >, std::allocator, std::allocator > const, int> > >::~unordered_multimap (this=0x7fa8089ddf80,
__in_chrg=) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/unordered_map.h:1245
#16 0x00007fa861f71d4a in arrow::StructType::Impl::~Impl (this=0x7fa8089ddf80, __in_chrg=) at /mnt/disk1/oap/emr_2024_09_23/_build/debug/CMake/resolve_dependency_modules/arrow/arrow_ep/src/arrow_ep/cpp/src/arrow/type.cc:1338
#17 0x00007fa861f71d70 in std::default_delete::operator() (this=0x7fa87698c358, __ptr=0x7fa8089ddf80) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/unique_ptr.h:95
#18 0x00007fa861f692b0 in std::unique_ptr >::~unique_ptr (this=0x7fa87698c358, __in_chrg=) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/unique_ptr.h:396
#19 0x00007fa861f51900 in arrow::StructType::~StructType (this=0x7fa87698c310, __in_chrg=) at /mnt/disk1/oap/emr_2024_09_23/_build/debug/CMake/resolve_dependency_modules/arrow/arrow_ep/src/arrow_ep/cpp/src/arrow/type.cc:1351
#20 0x00007fa861fa3ef0 in std::_Destroy (__pointer=0x7fa87698c310) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/stl_construct.h:151
#21 0x00007fa861fa39e0 in std::allocator_traits >::destroy (__p=0x7fa87698c310) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/alloc_traits.h:648
#22 0x00007fa861fa199d in std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x7fa87698c300) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/shared_ptr_base.h:613
#23 0x00007fa861653565 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use (this=0x7fa87698c300) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/shared_ptr_base.h:175
#24 0x00007fa8616523b8 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold (this=0x7fa87698c300) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/shared_ptr_base.h:199
#25 0x00007fa861650c98 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7fa87698c300) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/shared_ptr_base.h:353
#26 0x00007fa861652d0f in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7fa81437b8b0, __in_chrg=) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/shared_ptr_base.h:1071
#27 0x00007fa8616ea780 in std::__shared_ptr::~__shared_ptr (this=0x7fa81437b8a8, __in_chrg=) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/shared_ptr_base.h:1524
#28 0x00007fa8616ea79c in std::shared_ptr::~shared_ptr (this=0x7fa81437b8a8, __in_chrg=) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/shared_ptr.h:175
#29 0x00007fa861fab286 in arrow::(anonymous namespace)::SchemaImporter::~SchemaImporter (this=0x7fa81437b860, __in_chrg=)
at /mnt/disk1/oap/emr_2024_09_23/_build/debug/CMake/resolve_dependency_modules/arrow/arrow_ep/src/arrow_ep/cpp/src/arrow/c/bridge.cc:967
#30 0x00007fa861faee71 in arrow::ImportSchema (schema=0x7fa81437b980) at /mnt/disk1/oap/emr_2024_09_23/_build/debug/CMake/resolve_dependency_modules/arrow/arrow_ep/src/arrow_ep/cpp/src/arrow/c/bridge.cc:1425
#31 0x00007fa837c4cfa8 in gluten::toArrowSchema (rowType=std::shared_ptr (use count 2, weak count 0) = {...}, pool=0x7fa8768f1410) at /mnt/disk1/code/emr_2024_09_23/gluten/cpp/velox/utils/VeloxArrowUtils.cc:35
#32 0x00007fa837b258f8 in gluten::VeloxHashShuffleWriter::initColumnTypes (this=0x7fa8768c2200, rv=...) at /mnt/disk1/code/emr_2024_09_23/gluten/cpp/velox/shuffle/VeloxHashShuffleWriter.cc:732
#33 0x00007fa837b25fc2 in gluten::VeloxHashShuffleWriter::initFromRowVector (this=0x7fa8768c2200, rv=...) at /mnt/disk1/code/emr_2024_09_23/gluten/cpp/velox/shuffle/VeloxHashShuffleWriter.cc:801
#34 0x00007fa837b224c8 in gluten::VeloxHashShuffleWriter::partitioningAndDoSplit (this=0x7fa8768c2200, rv=std::shared_ptr (use count 2, weak count 0) = {...}, memLimit=502477619)
at /mnt/disk1/code/emr_2024_09_23/gluten/cpp/velox/shuffle/VeloxHashShuffleWriter.cc:317
#35 0x00007fa837b21e5e in gluten::VeloxHashShuffleWriter::write (this=0x7fa8768c2200, cb=std::shared_ptr (use count 4, weak count 0) = {...}, memLimit=502477619)
at /mnt/disk1/code/emr_2024_09_23/gluten/cpp/velox/shuffle/VeloxHashShuffleWriter.cc:300
#36 0x00007fa86168f7d7 in Java_org_apache_gluten_vectorized_ShuffleWriterJniWrapper_write (env=0x7fa895bb0ae8, wrapper=0x7fa81437c380, shuffleWriterHandle=317827579908, numRows=1118, batchHandle=446676598789, memLimit=502477619)
at /mnt/disk1/code/emr_2024_09_23/gluten/cpp/core/jni/JniWrapper.cc:1013
#37 0x00007fa8b2e18427 in ?? ()
#38 0x0000000000000000 in ?? ()
```

### Spark version

Spark-3.4.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.