apache / apache/arrow

[Python] SIGABRT: terminate called without an active exception (core dumped)

Open
#34,314 1 comment 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

# Description of the bug

Sometimes, the python executable crashes at the end of one of my company's scripts with SIGABRT and the following message to stderr: `terminate called without an active exception`

The coredump (374.5MB) showed that a parquet thread was responsible.

I do not have a minimal example, as the bug was hard to reproduce. The python script looked like this:

```
[very little python code]

df = pd.read_parquet(file) # only line using parquet

[little python code]

# last executed lines
with open(file_out, 'w+') as fd:
fd.write(output)
```

The script ran fully and worked well outside of the nonzero return code, stderr message, and core dump.

# Versions/Platform

Linux (happened both on `Debian 5.10.162-1 (2023-01-21) x86_64` and `Ubuntu 22.04.1`)
Python Version 3.9.2
pyarrow Version: 10.0.0

# Full Backtrace

It was obtained from the dumped core.

```
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f9c2a5b6537 in __GI_abort () at abort.c:79
#2 0x00007f9c1b0857ec in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007f9c1b090966 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007f9c1b0909d1 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007f9c1b0903cc in __gxx_personality_v0 () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007f9c278238a4 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#7 0x00007f9c27823f4e in _Unwind_ForcedUnwind () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#8 0x00007f9c2a917c30 in __GI___pthread_unwind (buf=) at unwind.c:121
#9 0x00007f9c2a90f18c in __do_cancel () at pthreadP.h:310
#10 __pthread_exit (value=) at pthread_exit.c:28
#11 0x0000000000645df5 in PyThread_exit_thread ()
#12 0x00000000004262ae in ?? ()
#13 0x00000000005327b2 in PyEval_RestoreThread ()
#14 0x0000000000609f83 in PyGILState_Ensure ()
#15 0x00007f9c1e205961 in arrow::py::PyBuffer::~PyBuffer() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_python.so.1000
#16 0x00007f9c1e206ea6 in std::_Sp_counted_ptr::_M_dispose() ()
from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_python.so.1000
#17 0x00007f9c1b71709a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#18 0x00007f9c1b719d7d in arrow::Future >::SetResult(arrow::Result >)::{lambda(void*)#1}::_FUN(void*) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#19 0x00007f9c1c1801a1 in arrow::ConcreteFutureImpl::~ConcreteFutureImpl() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#20 0x00007f9c1c205c5e in arrow::io::internal::ReadRangeCache::Impl::~Impl() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#21 0x00007f9c1da330ea in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
from /usr/local/lib/python3.9/dist-packages/pyarrow/libparquet.so.1000
#22 0x00007f9c1dad0f4e in parquet::ParquetFileReader::~ParquetFileReader() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libparquet.so.1000
#23 0x00007f9c1db3f970 in parquet::arrow::(anonymous namespace)::FileReaderImpl::~FileReaderImpl() ()
from /usr/local/lib/python3.9/dist-packages/pyarrow/libparquet.so.1000
#24 0x00007f9c1da330ea in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
from /usr/local/lib/python3.9/dist-packages/pyarrow/libparquet.so.1000
#25 0x00007f9c1db95633 in std::_Function_handler > ()> > (), parquet::arrow::RowGroupGenerator>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) ()
from /usr/local/lib/python3.9/dist-packages/pyarrow/libparquet.so.1000
#26 0x00007f9c1e0ac74e in std::_Sp_counted_ptr_inplace >::State, std::allocator >::State>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#27 0x00007f9c1da330ea in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libparquet.so.1000
#28 0x00007f9c1db92200 in std::_Function_handler > (), arrow::MergedGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libparquet.so.1000
#29 0x00007f9c1e085f09 in std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#30 0x00007f9c1e50db5a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /usr/local/lib/python3.9/dist-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#31 0x00007f9c1e0876a0 in std::_Function_handler > (), arrow::dataset::SlicingGenerator>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#32 0x00007f9c1e07e33e in std::_Sp_counted_ptr_inplace >::State, std::allocator >::State>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#33 0x00007f9c1e50db5a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /usr/local/lib/python3.9/dist-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#34 0x00007f9c1e086ac0 in std::_Function_handler > (), arrow::SerialReadaheadGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#35 0x00007f9c1e089156 in arrow::Future > ()> >::SetResult(arrow::Result > ()> >)::{lambda(void*)#1}::_FUN(void*) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#36 0x00007f9c1c1801a1 in arrow::ConcreteFutureImpl::~ConcreteFutureImpl() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#37 0x00007f9c1e50db5a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /usr/local/lib/python3.9/dist-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#38 0x00007f9c1e50db5a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /usr/local/lib/python3.9/dist-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#39 0x00007f9c1e086b60 in std::_Function_handler > (), arrow::FutureFirstGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#40 0x00007f9c1e085eb9 in std::_Sp_counted_ptr_inplace >::State, std::allocator >::State>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#41 0x00007f9c1e50db5a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /usr/local/lib/python3.9/dist-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#42 0x00007f9c1e087560 in std::_Function_handler > (), arrow::DefaultIfEmptyGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#43 0x00007f9c1e085e89 in std::_Sp_counted_ptr_inplace >::State, std::allocator >::State>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#44 0x00007f9c1e50db5a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /usr/local/lib/python3.9/dist-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#45 0x00007f9c1e0874c0 in std::_Function_handler > > (), arrow::EnumeratingGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#46 0x00007f9c1e088f76 in arrow::Future > > ()> >::SetResult(arrow::Result > > ()> >)::{lambda(void*)#1}::_FUN(void*) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#47 0x00007f9c1c1801a1 in arrow::ConcreteFutureImpl::~ConcreteFutureImpl() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#48 0x00007f9c1e50db5a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /usr/local/lib/python3.9/dist-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#49 0x00007f9c1e50db5a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /usr/local/lib/python3.9/dist-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#50 0x00007f9c1e087420 in std::_Function_handler > > (), arrow::FutureFirstGenerator > > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#51 0x00007f9c1e0a90ad in std::_Sp_counted_ptr_inplace >, arrow::dataset::EnumeratedRecordBatch>::State, std::allocator >, arrow::dataset::EnumeratedRecordBatch>::State>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#52 0x00007f9c1e50db5a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /usr/local/lib/python3.9/dist-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#53 0x00007f9c1e087380 in std::_Function_handler (), arrow::MappingGenerator >, arrow::dataset::EnumeratedRecordBatch> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#54 0x00007f9c1e0ab106 in std::_Sp_counted_ptr_inplace::State, std::allocator::State>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#55 0x00007f9c1e50db5a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /usr/local/lib/python3.9/dist-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#56 0x00007f9c1e086de0 in std::_Function_handler (), arrow::MergedGenerator >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#57 0x00007f9c1e0a8fb2 in std::_Sp_counted_ptr_inplace::State, std::allocator::State>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#58 0x00007f9c1e50db5a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /usr/local/lib/python3.9/dist-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#59 0x00007f9c1e086d40 in std::_Function_handler (), arrow::ReadaheadGenerator >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#60 0x00007f9c1e0ab3bd in std::_Sp_counted_ptr_inplace >::State, std::allocator >::State>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#61 0x00007f9c1e50db5a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /usr/local/lib/python3.9/dist-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#62 0x00007f9c1e086ca0 in std::_Function_handler > (), arrow::MappingGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#63 0x00007f9c1bfbbf4f in arrow::compute::(anonymous namespace)::SourceNode::~SourceNode() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#64 0x00007f9c1c047d4f in arrow::compute::(anonymous namespace)::ExecPlanImpl::~ExecPlanImpl() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#65 0x00007f9c1c049842 in std::_Sp_counted_ptr::_M_dispose() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#66 0x00007f9c1e50db5a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /usr/local/lib/python3.9/dist-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#67 0x00007f9c1e0536ca in arrow::internal::FnOnce::FnImpl::WrapStatusyOnComplete::Callback >::~FnImpl() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_dataset.so.1000
#68 0x00007f9c1c184877 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#69 0x00007f9c1e204054 in void arrow::Future::MarkFinished(arrow::Status) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_python.so.1000
#70 0x00007f9c1c028ce5 in arrow::internal::FnOnce::FnImpl::WrapStatusyOnComplete::Callback >::invoke(arrow::FutureImpl const&) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#71 0x00007f9c1c184863 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#72 0x00007f9c1e204054 in void arrow::Future::MarkFinished(arrow::Status) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_python.so.1000
#73 0x00007f9c1c1c07b6 in arrow::util::(anonymous namespace)::AsyncTaskSchedulerImpl::OnTaskFinished(arrow::Status const&, int) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#74 0x00007f9c1c184863 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#75 0x00007f9c1e204054 in void arrow::Future::MarkFinished(arrow::Status) () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow_python.so.1000
#76 0x00007f9c1c04e69d in arrow::internal::FnOnce::FnImpl, std::function)> >::invoke() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#77 0x00007f9c1c1689e5 in arrow::internal::FnOnce::operator()() && () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#78 0x00007f9c1c1636d8 in std::thread::_State_impl > >::_M_run() () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#79 0x00007f9c1ce88090 in execute_native_thread_routine () from /usr/local/lib/python3.9/dist-packages/pyarrow/libarrow.so.1000
#80 0x00007f9c2a90dea7 in start_thread (arg=) at pthread_create.c:477
#81 0x00007f9c2a690a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
```

# Workaround

Adding
`import pyarrow.parquet as pq`
and replacing
`df = pd.read_parquet(file) # only line using parquet`
by
`df = pq.read_table(file, use_threads=False).to_pandas(use_threads=False)`
made the bug disappear, but increased the reading time.

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Research direction

Start with the reported pd.read_parquet call, pyarrow 10.0.0 environment, and the provided backtrace through parquet, Arrow futures, and PyBuffer destruction. Reproduction is not supplied, so first establish a minimal reproducible case; done would mean identifying the SIGABRT cause and validating a fix with a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.