[Python] Interpreter hangs at exit after `pq.read_table(python_file_object)` - thread pool teardown deadlock
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
### Describe the bug, including details regarding any error messages, version, and platform.
Reading a parquet file by passing a **Python file object** to `pyarrow.parquet.read_table`
makes the interpreter hang forever at shutdown on my and other similar machines:
```python
# repro.py - hangs at exit, every run
import pyarrow.parquet as pq
with open("any.parquet", "rb") as f:
pq.read_table(f)
print("done, exiting...") # prints - then the process never exits
```
```
$ python repro.py
done, exiting...
# hangs forever; SIGKILL required
```
Passing a **path** instead of a file object does not hang:
```python
pq.read_table("any.parquet") # exits cleanly, 8/8 runs
```
`use_threads=False` also avoids the hang even with a file object (8/8 clean), as does
`pa.set_cpu_count(1)`, so this looks like a CPU thread-pool teardown deadlock that is
only triggered when the read went through a Python file handle (GIL interaction during
pool shutdown?).
#### How we found it
`pandas.read_parquet` (which internally opens the file and passes a handle to pyarrow)
was hanging **intermittently** (~50% of runs when reading 3 files, rarer with 1 file).
Bisecting pandas out of the equation led to the deterministic file-object reproducer
above. So this same bug likely explains "flaky CI hang" reports from pandas users.
#### Details
- The script body completes and **`atexit` handlers run**; the hang is after that,
during interpreter/C-level finalization. `faulthandler` (SIGABRT) produces no dump.
- The file content doesn't matter - reproduces with a 5-row single-column file.
- Closing the file object (or not) before exit makes no difference.
- A hung process shows the main thread plus ~24 CPU-pool threads all in
`futex_do_wait` (from `/proc//task/*/wchan`), and a `jemalloc_bg_thd`:
```
python(futex_do_wait) jemalloc_bg_thd(futex_do_wait) python(futex_do_wait) x24 ...
```
#### Version matrix (same reproducer, 4 runs each; `124` = hung/timeout, `0` = clean exit)
| pyarrow | result |
|---|---|
| 25.0.0 | 124 124 124 124 |
| 24.0.0 | 124 124 124 124 |
| 23.0.1 | 124 124 124 124 |
| 22.0.0 | 124 0 124 124 |
| 25.0.0 + `use_threads=False` | 0 0 0 0 |
| 25.0.0 + path instead of file object | 0 0 0 0 |
(22.0.0 being intermittent rather than deterministic suggests a timing-dependent race
that became reliably lost at some point.)
#### Second interpreter build
Also reproduces on the distro CPython (Python 3.14.4, GCC 15.2.0 build,
`/usr/bin/python3.14`, fresh venv, pyarrow 25.0.0 wheel) - intermittent there
(3/6 runs hang) instead of deterministic, path-based reads clean 6/6. So this is
not specific to one interpreter build; build/timing differences only shift how
often the race is lost.
#### Environment
- pyarrow 25.0.0 (manylinux wheel from PyPI, installed via `uv`)
- Python 3.13.13 (python-build-standalone, Clang 22.1.3) - deterministic hang
- Python 3.14.4 (distro build, GCC 15.2.0) - intermittent hang (~50%)
- Linux 7.0.0-22-generic x86_64, glibc 2.43
- 24 CPUs
- pandas 3.0.3 (only for the original discovery path; not needed for the reproducer)
- Persists after OS upgrade + reboot (kernel 7.0.0-22 → 7.0.0-27, glibc 2.43-2ubuntu2)
### Component(s)
Python
### Component(s)
Python
Contributor guide
Research direction
Run the repro.py script with a Python file object, then compare the path and use_threads=False cases described in the report. Start at pyarrow.parquet.read_table and trace the file-object path through CPU thread-pool teardown; done means the file-object case exits cleanly with threads enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100