apache / apache/arrow

pq.read_table doesn't seem to be thread safe

Open
#33,189 6 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Before PR: gets merged in master, I am using multithreading to improve read bandwidth from S3. Even after that PR gets merged, I probably will still try to use multithreading to some extent.

However pq.read_table from S3 doesn't seem to be thread safe. Seems like it uses the new dataset reader under the hood. I cannot provide a reproduction, not a stable one anyway. But this is roughly the script I have been using 

~~~

def get_next_batch(self, mapper_id, pos=None):

def download(file):
    return pq.read_table("s3://" + self.bucket + "/" + file, columns=self.columns, filters=self.filters)
 
executor = concurrent.futures.ThreadPoolExecutor(max_workers=self.workers)

futures= \{executor.submit(download, file): file for file in my_files}
for future inconcurrent.futures.as_completed(futures):
    yield  future.result()
~~~
The errors all have to do with malloc segfaults which makes me suspect the connection object is being reused across different pq.read_table invocations in different threads
```
(InputReaderNode pid=25001, ip=172.31.60.29) malloc_consolidate(): invalid chunk size
(InputReaderNode pid=25001, ip=172.31.60.29) \*\*\* SIGABRT received at time=1665464922 on cpu 9 \*\*\*
(InputReaderNode pid=25001, ip=172.31.60.29) PC: @     0x7f9a480a803b  (unknown)  raise
(InputReaderNode pid=25001, ip=172.31.60.29)     @     0x7f9a480a80c0       4160  (unknown)
(InputReaderNode pid=25001, ip=172.31.60.29)     @     0x7f9a480fa32c  (unknown)  (unknown)
```
Note, this multithreaded code is running inside a Ray actor process, but that shouldn't be a problem.

**Reporter**: [Ziheng Wang](https://issues.apache.org/jira/browse/ARROW-17984) / @marsupialtail
#### Original Issue Attachments:
- [_usr_bin_python3.8.1000.crash](https://issues.apache.org/jira/secure/attachment/13051570/_usr_bin_python3.8.1000.crash)

**Note**: *This issue was originally created as [ARROW-17984](https://issues.apache.org/jira/browse/ARROW-17984). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the Python pq.read_table entry point and the supplied ThreadPoolExecutor script, then inspect the dataset reader and S3 connection handling. First establish a stable concurrent-read reproduction; done means concurrent S3 reads no longer trigger the reported malloc failures and the behavior is covered by regression testing.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.