[Python][S3] Segmentation fault when running multithreading in Docker
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
When running such a simple test in Docker I'm getting `Segmentation fault` with PyArrow v14.0.2. The same test works fine in a versions bellow.
**Sample test**:
```python
import concurrent.futures
import pyarrow.parquet as pq
from time import time
feeds = [
"file1",
"file2",
...
"fileN",
]
def try_feed_load(feed: str):
start = time()
pq.read_table(f"s3://bucket_name/{feed}", filters=None)
end = time()
print(f"{feed} took {end - start} seconds")
with concurrent.futures.ThreadPoolExecutor(max_workers=6) as pool_executor:
for collected_data in pool_executor.map(try_feed_load, feeds):
pass
```
**To repeat**:
1. Start a container:
`docker run --rm -it --init --ulimit core=-1 --mount type=bind,source=/tmp/,target=/tmp/ -v /root/work:/work --entrypoint /bin/bash amd64/python:3.11-slim`
2. Install most recent PyArrow package:
```shell
cd /work
python3 -m venv .venv_recent
. .venv_recent/bin/activate
pip install pyarrow # That will install pyarrow-14.0.2-cp311-cp311-manylinux_2_28_x86_64.whl.metadata
```
3. Run a test. It fails immediately:
```shell
python test2.py
Segmentation fault (core dumped)
```
**Workarounds**:
1. Change the script to run at most 1 worker
2. Downgrade to PyArrow v14.0.1
**Failure details**:
```gdb
Thread 67 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f8bf97956c0 (LWP 943)]
0x00007f8ca273cd50 in Aws::Utils::Threading::ReaderWriterLock::LockReader() () from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
(gdb) bt
#0 0x00007f8ca273cd50 in Aws::Utils::Threading::ReaderWriterLock::LockReader() () from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#1 0x00007f8ca275ee62 in Aws::Config::ConfigAndCredentialsCacheManager::GetConfigProfile(std::__cxx11::basic_string, std::allocator > const&) const
() from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#2 0x00007f8ca275f103 in Aws::Config::GetCachedConfigProfile(std::__cxx11::basic_string, std::allocator > const&) ()
from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#3 0x00007f8ca2761736 in Aws::Auth::STSAssumeRoleWebIdentityCredentialsProvider::STSAssumeRoleWebIdentityCredentialsProvider() ()
from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#4 0x00007f8ca2763866 in Aws::Auth::DefaultAWSCredentialsProviderChain::DefaultAWSCredentialsProviderChain() () from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#5 0x00007f8ca1fbf579 in arrow::fs::S3Options::ConfigureDefaultCredentials() () from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#6 0x00007f8ca1fceac0 in arrow::fs::S3Options::FromUri(arrow::internal::Uri const&, std::__cxx11::basic_string, std::allocator >*) ()
from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#7 0x00007f8ca12fa070 in arrow::fs::(anonymous namespace)::FileSystemFromUriReal(arrow::internal::Uri const&, std::__cxx11::basic_string, std::allocator > const&, arrow::io::IOContext const&, std::__cxx11::basic_string, std::allocator >*) ()
from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#8 0x00007f8ca12faa80 in arrow::fs::FileSystemFromUri(std::__cxx11::basic_string, std::allocator > const&, arrow::io::IOContext const&, std::__cxx11::basic_string, std::allocator >*) () from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#9 0x00007f8ca12fab7c in arrow::fs::FileSystemFromUriOrPath(std::__cxx11::basic_string, std::allocator > const&, arrow::io::IOContext const&, std::__cxx11::basic_string, std::allocator >*) () from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#10 0x00007f8ca12facf3 in arrow::fs::FileSystemFromUriOrPath(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator >*) () from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#11 0x00007f8bfa118712 in __pyx_pf_7pyarrow_3_fs_10FileSystem_2from_uri(_object*) () from /usr/local/lib/python3.11/site-packages/pyarrow/_fs.cpython-311-x86_64-linux-gnu.so
#12 0x00007f8ca50d5843 in PyObject_Vectorcall () from /usr/local/bin/../lib/libpython3.11.so.1.0
#13 0x00007f8ca50ca92b in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.11.so.1.0
#14 0x00007f8ca50c94ba in ?? () from /usr/local/bin/../lib/libpython3.11.so.1.0
#15 0x00007f8ca50c4a5d in _PyObject_FastCallDictTstate () from /usr/local/bin/../lib/libpython3.11.so.1.0
#16 0x00007f8ca50e6244 in ?? () from /usr/local/bin/../lib/libpython3.11.so.1.0
#17 0x00007f8ca50c3721 in ?? () from /usr/local/bin/../lib/libpython3.11.so.1.0
#18 0x00007f8ca50c35cf in _PyObject_MakeTpCall () from /usr/local/bin/../lib/libpython3.11.so.1.0
#19 0x00007f8ca50ca92b in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.11.so.1.0
#20 0x00007f8ca50c94ba in ?? () from /usr/local/bin/../lib/libpython3.11.so.1.0
#21 0x00007f8ca50ccab1 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.11.so.1.0
#22 0x00007f8ca50c94ba in ?? () from /usr/local/bin/../lib/libpython3.11.so.1.0
#23 0x00007f8ca50ccab1 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.11.so.1.0
#24 0x00007f8ca50c94ba in ?? () from /usr/local/bin/../lib/libpython3.11.so.1.0
#25 0x00007f8ca50f2e6d in ?? () from /usr/local/bin/../lib/libpython3.11.so.1.0
#26 0x00007f8ca516c3c4 in ?? () from /usr/local/bin/../lib/libpython3.11.so.1.0
#27 0x00007f8ca516c354 in ?? () from /usr/local/bin/../lib/libpython3.11.so.1.0
#28 0x00007f8ca4d72044 in start_thread (arg=) at ./nptl/pthread_create.c:442
#29 0x00007f8ca4df1880 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
(gdb)
```
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.