[Python] Import of pyarrow.parquet and loading of non-existing file threw exception with incompatible pandas
- Lingua principale
- C++
- Stelle
- 17.1k
- Fork
- 4.3k
- Merge medio
- 3g 20h
- PR unite (30g)
- 94
Descrizione
### Describe the bug, including details regarding any error messages, version, and platform.
I was doing something like this:
```python
import numpy as np
import pyarrow.parquet as pq
tab = pq.read_table("non-existing.parquet")
```
This failed with:
```
tab = pq.read_table("train-00000-of-00004.parquet")
File "/home/inferencer/.local/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 1793, in read_table
dataset = ParquetDataset(
File "/home/inferencer/.local/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 1291, in __init__
import pyarrow.dataset as ds
File "/home/inferencer/.local/lib/python3.10/site-packages/pyarrow/dataset.py", line 24, in
from pyarrow._dataset import ( # noqa
File "pyarrow/_dataset.pyx", line 159, in init pyarrow._dataset
File "pyarrow/_compute.pyx", line 2715, in pyarrow._compute.Expression._scalar
File "pyarrow/scalar.pxi", line 1277, in pyarrow.lib.scalar
File "pyarrow/array.pxi", line 57, in pyarrow.lib._is_array_like
File "pyarrow/pandas-shim.pxi", line 126, in pyarrow.lib._PandasAPIShim._have_pandas_internal File "pyarrow/types.pxi", line 105, in pyarrow.lib._datatype_to_pep3118
File "pyarrow/pandas-shim.pxi", line 108, in pyarrow.lib._PandasAPIShim._check_import
File "pyarrow/pandas-shim.pxi", line 50, in pyarrow.lib._PandasAPIShim._import_pandas
File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/__init__.py", line 22, in
from pandas.compat import is_numpy_dev as _is_numpy_dev
File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/compat/__init__.py", line 15, in
from pandas.compat.numpy import ( File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/compat/numpy/__init__.py", line 4, in
from pandas.util.version import Version File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/util/__init__.py", line 1, in
from pandas.util._decorators import ( # noqa:F401 File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/util/_decorators.py", line 14, in
from pandas._libs.properties import cache_readonly # noqa:F401 File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/_libs/__init__.py", line 13, in
from pandas._libs.interval import Interval
File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
```
This is quite strange, because I hoped that pyarrow does not depend on pandas / pandas installed correctly. And it was strange that this threw only at `pq.read_table` call, and even before checking if file existed.
I thought that pandas depended on pyarrow and not in reverse. Is it not true? And in any case, some explicit error message with minimal required pandas version might be more clear to the user.
This was with: `pandas 1.4.3 and pyarrow-19.0.1`
After I did `python -m pip uinstall pyarrow pandas` and reinstalled again (got `pandas-2.2.3 pyarrow-19.0.1`), the problem fixed itself, and became to error-out at the non-existing file.
Thanks!
### Component(s)
Python
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Parti dal punto di ingresso Python per parquet `pyarrow/parquet/core.py` e segui `read_table` nel percorso di importazione di `ParquetDataset` e `pyarrow.dataset` mostrato nel traceback. Riproduci il problema con pandas 1.4.3 e pyarrow 19.0.1, quindi confronta con pandas 2.2.3; il lavoro è completato quando il caso di pandas incompatibile produce un errore chiaro e intenzionale oppure si evita che fallisca prima del controllo del file mancante.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- pandas, python
- Ambito
- data
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 35/100