apache / apache/arrow

[C++][Parquet] Thread crash at exit in Arrow-15.0.0 dataset scan

Open
#39,862 4 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

In Arrow-15.0.0 when using a threaded dataset scan I sometimes see the following crash at exit:
```
#1 0x00007ffff691e351 in arrow::Status arrow::internal::Executor::Spawn const&, arrow::FutureImpl::CallbackRecord&&, bool)::{lambda()#1}>(arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool)::{lambda()#1}&&) () from /jump/software/rhel8/apache-arrow-15.0.0-cxx20-gcc10/lib64/libarrow.so.1500
#2 0x00007ffff691e6ee in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool) ()
from /jump/software/rhel8/apache-arrow-15.0.0-cxx20-gcc10/lib64/libarrow.so.1500
#3 0x00007ffff691e99d in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) ()
from /jump/software/rhel8/apache-arrow-15.0.0-cxx20-gcc10/lib64/libarrow.so.1500
#4 0x00007ffff68c7454 in void arrow::Future::MarkFinished(arrow::Status) ()
from /jump/software/rhel8/apache-arrow-15.0.0-cxx20-gcc10/lib64/libarrow.so.1500
#5 0x00007ffff691bf1e in arrow::internal::FnOnce::FnImpl::WrapStatusyOnComplete::Callback, std::allocator > > const&)::{lambda(arrow::Status const&)#1}> >::invoke(arrow::FutureImpl const&) () from /jump/software/rhel8/apache-arrow-15.0.0-cxx20-gcc10/lib64/libarrow.so.1500
#6 0x00007ffff691e65c in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool) ()
from /jump/software/rhel8/apache-arrow-15.0.0-cxx20-gcc10/lib64/libarrow.so.1500
#7 0x00007ffff691e99d in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) ()
from /jump/software/rhel8/apache-arrow-15.0.0-cxx20-gcc10/lib64/libarrow.so.1500
#8 0x00007ffff68c0506 in arrow::internal::FnOnce::FnImpl >, arrow::io::RandomAccessFile::ReadAsync(arrow::io::IOContext const&, long, long)::{lambda()#1})> >::invoke() ()
from /jump/software/rhel8/apache-arrow-15.0.0-cxx20-gcc10/lib64/libarrow.so.1500
#9 0x00007ffff694d6c9 in std::thread::_State_impl > >::_M_run() () from /jump/software/rhel8/apache-arrow-15.0.0-cxx20-gcc10/lib64/libarrow.so.1500
#10 0x00007ffff5190640 in std::execute_native_thread_routine (__p=0x777df0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#11 0x00007ffff17021cf in start_thread () from /lib64/libpthread.so.0
#12 0x00007ffff4794dd3 in clone () from /lib64/libc.so.6
```

The crash is non-deterministic, happens around 50% of the time on the machine I'm testing on. This problem only appears when the program terminates immediately after the end of the scan; my guess would be that the threads are not being cancelled/joined correctly and so are still running when the program terminates.

To create the input data:
```
import pyarrow.parquet as pq
import pyarrow as pa
pq.write_table(pa.Table.from_pandas(pd.read_parquet("http://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2023-01.parquet")), "taxi.parquet")
```
For some reason, using the downloaded Parquet directly does NOT show the problem.

Then to produce the crash:
```
#include
#include

int
main ()
{
const auto format = std::make_shared ();
const auto options = arrow::dataset::FileSystemFactoryOptions {};
const std::shared_ptr filesystem =
std::make_shared ();

std::vector object_ids { "taxi.parquet" };

auto factory = arrow::dataset::FileSystemDatasetFactory::Make (
filesystem, std::move (object_ids), format, options)
.ValueOrDie ();
const auto full_dataset = factory->Finish ().ValueOrDie ();

auto builder = full_dataset->NewScan ().ValueOrDie ();
(void)builder->UseThreads (true);
(void)builder->Project ({ "passenger_count", "trip_distance" });
auto scanner = builder->Finish ().ValueOrDie ();
(void)scanner->Head (10);
}
```
I compiled this with:
```
g++ -std=c++20 arrow_bug.cpp $(pkg-config arrow --cflags --libs) $(pkg-config arrow-dataset --cflags --libs)
```

I've also seen this crash with 14.0.1, but can't reproduce on 12.0.0, so I imagine something changed between them.

### Component(s)

C++, Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.