apache / apache/arrow

[Python] `pyarrow.write_feather` can't be used in `atexit` contexts to write a `pandas.DataFrame`

Open
#41,815 1 comment 0 reactions 1 assignee Claimed by @pjh40 View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 20h
Merged PRs (30d)
94

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When `pyarrow.write_feather()` is given a `pandas.DataFrame`, `write_feather()` unconditionally calls `Table.from_pandas()` with the default `nthreads=None` argument. This is then passed to `pandas_compat.dataframe_to_arrays()`, allowing it to heuristically use a `concurrent.futures.ThreadPoolExecuter` to convert columns. This causes a runtime error when `write_feather` is used in an `atexit` (or `weakref.finalize`) context on exit of the interpreter:
```
RuntimeError: cannot schedule new futures after interpreter shutdown
```
This scenario can be avoided by adding a `use_threads` parameter to `write_feather` that can be used to force serial operation.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.