[Python] Import of pyarrow.parquet and loading of non-existing file threw exception with incompatible pandas
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I was doing something like this:
```python
import numpy as np
import pyarrow.parquet as pq
tab = pq.read_table("non-existing.parquet")
```
This failed with:
```
tab = pq.read_table("train-00000-of-00004.parquet")
File "/home/inferencer/.local/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 1793, in read_table
dataset = ParquetDataset(
File "/home/inferencer/.local/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 1291, in __init__
import pyarrow.dataset as ds
File "/home/inferencer/.local/lib/python3.10/site-packages/pyarrow/dataset.py", line 24, in
from pyarrow._dataset import ( # noqa
File "pyarrow/_dataset.pyx", line 159, in init pyarrow._dataset
File "pyarrow/_compute.pyx", line 2715, in pyarrow._compute.Expression._scalar
File "pyarrow/scalar.pxi", line 1277, in pyarrow.lib.scalar
File "pyarrow/array.pxi", line 57, in pyarrow.lib._is_array_like
File "pyarrow/pandas-shim.pxi", line 126, in pyarrow.lib._PandasAPIShim._have_pandas_internal File "pyarrow/types.pxi", line 105, in pyarrow.lib._datatype_to_pep3118
File "pyarrow/pandas-shim.pxi", line 108, in pyarrow.lib._PandasAPIShim._check_import
File "pyarrow/pandas-shim.pxi", line 50, in pyarrow.lib._PandasAPIShim._import_pandas
File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/__init__.py", line 22, in
from pandas.compat import is_numpy_dev as _is_numpy_dev
File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/compat/__init__.py", line 15, in
from pandas.compat.numpy import ( File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/compat/numpy/__init__.py", line 4, in
from pandas.util.version import Version File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/util/__init__.py", line 1, in
from pandas.util._decorators import ( # noqa:F401 File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/util/_decorators.py", line 14, in
from pandas._libs.properties import cache_readonly # noqa:F401 File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/_libs/__init__.py", line 13, in
from pandas._libs.interval import Interval
File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
```
This is quite strange, because I hoped that pyarrow does not depend on pandas / pandas installed correctly. And it was strange that this threw only at `pq.read_table` call, and even before checking if file existed.
I thought that pandas depended on pyarrow and not in reverse. Is it not true? And in any case, some explicit error message with minimal required pandas version might be more clear to the user.
This was with: `pandas 1.4.3 and pyarrow-19.0.1`
After I did `python -m pip uinstall pyarrow pandas` and reinstalled again (got `pandas-2.2.3 pyarrow-19.0.1`), the problem fixed itself, and became to error-out at the non-existing file.
Thanks!
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.