apache / apache/arrow

[Python] to_table() call on dataset gets stuck and ignores timeout threshold

Open
#40,019 0 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

We're seeing an intermittent behaviour where a call to the `to_table()` method can get stuck when being called on a dataset stored in S3. The stuck call doesn't return before we give up after a couple of hours. But very often a subsequent request succeeds on the same dataset.

We have a python stack trace pointing at that specific method as being where its getting stuck, but we haven't got a way to see where in the C++ code it is stuck.

Our system has a couple of mechanisms to recover from processes that aren't making progress in the expected timeframe. The preferred and first attempt issues a sigalarm to interrupt the process. However the C-extension appears to be blocking this approach and isn't working. We revert then to an alternative approach that kills the container the process is running in.

It's not totally clear where the problem here lies, although a likely candidate is attempting to fetch the dataset from S3. However if this is the cause of the problem, we are confused why the request to S3 isn't timing out after a few minutes.

Version info:
pyarrow version 14.0.2
python 3.11

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.