pq.read_table doesn't seem to be thread safe
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
Before PR: gets merged in master, I am using multithreading to improve read bandwidth from S3. Even after that PR gets merged, I probably will still try to use multithreading to some extent.
However pq.read_table from S3 doesn't seem to be thread safe. Seems like it uses the new dataset reader under the hood. I cannot provide a reproduction, not a stable one anyway. But this is roughly the script I have been using
~~~
def get_next_batch(self, mapper_id, pos=None):
def download(file):
return pq.read_table("s3://" + self.bucket + "/" + file, columns=self.columns, filters=self.filters)
executor = concurrent.futures.ThreadPoolExecutor(max_workers=self.workers)
futures= \{executor.submit(download, file): file for file in my_files}
for future inconcurrent.futures.as_completed(futures):
yield future.result()
~~~
The errors all have to do with malloc segfaults which makes me suspect the connection object is being reused across different pq.read_table invocations in different threads
```
(InputReaderNode pid=25001, ip=172.31.60.29) malloc_consolidate(): invalid chunk size
(InputReaderNode pid=25001, ip=172.31.60.29) \*\*\* SIGABRT received at time=1665464922 on cpu 9 \*\*\*
(InputReaderNode pid=25001, ip=172.31.60.29) PC: @ 0x7f9a480a803b (unknown) raise
(InputReaderNode pid=25001, ip=172.31.60.29) @ 0x7f9a480a80c0 4160 (unknown)
(InputReaderNode pid=25001, ip=172.31.60.29) @ 0x7f9a480fa32c (unknown) (unknown)
```
Note, this multithreaded code is running inside a Ray actor process, but that shouldn't be a problem.
**Reporter**: [Ziheng Wang](https://issues.apache.org/jira/browse/ARROW-17984) / @marsupialtail
#### Original Issue Attachments:
- [_usr_bin_python3.8.1000.crash](https://issues.apache.org/jira/secure/attachment/13051570/_usr_bin_python3.8.1000.crash)
**Note**: *This issue was originally created as [ARROW-17984](https://issues.apache.org/jira/browse/ARROW-17984). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with the Python pq.read_table entry point and the supplied ThreadPoolExecutor script, then inspect the dataset reader and S3 connection handling. First establish a stable concurrent-read reproduction; done means concurrent S3 reads no longer trigger the reported malloc failures and the behavior is covered by regression testing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100