[Python] `pyarrow.write_feather` can't be used in `atexit` contexts to write a `pandas.DataFrame`
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 94
Description
### Describe the bug, including details regarding any error messages, version, and platform.
When `pyarrow.write_feather()` is given a `pandas.DataFrame`, `write_feather()` unconditionally calls `Table.from_pandas()` with the default `nthreads=None` argument. This is then passed to `pandas_compat.dataframe_to_arrays()`, allowing it to heuristically use a `concurrent.futures.ThreadPoolExecuter` to convert columns. This causes a runtime error when `write_feather` is used in an `atexit` (or `weakref.finalize`) context on exit of the interpreter:
```
RuntimeError: cannot schedule new futures after interpreter shutdown
```
This scenario can be avoided by adding a `use_threads` parameter to `write_feather` that can be used to force serial operation.
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.